Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 07:57:51.505050 398 test_util.cc:268] Using random seed: -673058173
I20240501 07:57:51.511852 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:57:51.511969 398 ts_itest-base.cc:113] --------------
I20240501 07:57:51.512027 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:57:51.512050 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:57:51.512068 398 ts_itest-base.cc:116] --------------
2024-05-01T07:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:57:51Z Disabled control of system clock
I20240501 07:57:51.521044 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44595
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44595 with env {}
W20240501 07:57:51.626451 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:51.626657 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:51.626695 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:51.629635 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:57:51.629693 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:51.629711 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:57:51.629730 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:57:51.633878 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44595
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:51.634922 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:51.636065 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:51.644487 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:51.645251 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:51.646044 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:51.646914 410 server_base.cc:1034] running on GCE node
I20240501 07:57:51.647481 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:51.648397 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:51.649552 410 hybrid_clock.cc:648] HybridClock initialized: now 1714550271649507 us; error 58 us; skew 500 ppm
I20240501 07:57:51.651489 410 webserver.cc:460] Webserver started at http://127.0.99.190:37811/ using document root <none> and password file <none>
I20240501 07:57:51.652029 410 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:51.652089 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:51.652256 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:51.653862 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8baa8037722d47b181c78f618883cada"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.654327 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8baa8037722d47b181c78f618883cada"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.657712 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:57:51.659929 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:51.660893 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:57:51.661013 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8baa8037722d47b181c78f618883cada"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.661100 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:51.673638 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:51.674285 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:51.674456 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:51.680965 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44595
I20240501 07:57:51.681044 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44595 every 8 connection(s)
I20240501 07:57:51.681915 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:57:51.684495 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:51.686450 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 410
I20240501 07:57:51.686591 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:57:51.689440 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada: Bootstrap starting.
I20240501 07:57:51.691879 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:51.692682 477 log.cc:824] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:51.694401 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada: No bootstrap required, opened a new log
I20240501 07:57:51.697351 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } }
I20240501 07:57:51.697584 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:51.697646 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa8037722d47b181c78f618883cada, State: Initialized, Role: FOLLOWER
I20240501 07:57:51.698104 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } }
I20240501 07:57:51.698225 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:57:51.698314 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:57:51.698423 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:51.699299 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } }
I20240501 07:57:51.699641 477 leader_election.cc:302] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8baa8037722d47b181c78f618883cada; no voters:
I20240501 07:57:51.699923 477 leader_election.cc:288] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:57:51.700417 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:51.700611 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [term 1 LEADER]: Becoming Leader. State: Replica: 8baa8037722d47b181c78f618883cada, State: Running, Role: LEADER
I20240501 07:57:51.700896 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } }
I20240501 07:57:51.701110 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:57:51.702503 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8baa8037722d47b181c78f618883cada. Latest consensus state: current_term: 1 leader_uuid: "8baa8037722d47b181c78f618883cada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } } }
I20240501 07:57:51.702541 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8baa8037722d47b181c78f618883cada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa8037722d47b181c78f618883cada" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44595 } } }
I20240501 07:57:51.702714 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [sys.catalog]: This master's current role is: LEADER
I20240501 07:57:51.702703 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada [sys.catalog]: This master's current role is: LEADER
I20240501 07:57:51.703243 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:57:51.705854 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:57:51.710258 493 catalog_manager.cc:1347] Generated new cluster ID: e7512d08e7d74ec697afe49f0e48018d
I20240501 07:57:51.710325 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:57:51.746606 493 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:57:51.747813 493 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:57:51.753836 493 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8baa8037722d47b181c78f618883cada: Generated new TSK 0
I20240501 07:57:51.754525 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:57:51.760251 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 07:57:51.864180 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:51.864511 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:51.864590 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:51.867710 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:51.867820 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:57:51.871371 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:51.872200 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:51.873245 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:51.881647 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:51.882496 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:51.883253 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:51.884187 501 server_base.cc:1034] running on GCE node
I20240501 07:57:51.884557 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:51.885092 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:51.886240 501 hybrid_clock.cc:648] HybridClock initialized: now 1714550271886220 us; error 35 us; skew 500 ppm
I20240501 07:57:51.888358 501 webserver.cc:460] Webserver started at http://127.0.99.129:34567/ using document root <none> and password file <none>
I20240501 07:57:51.888952 501 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:51.889017 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:51.889205 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:51.890931 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6d4c44480ae94a3fafe4c6272b378009"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.891484 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6d4c44480ae94a3fafe4c6272b378009"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.894901 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:57:51.897193 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:51.898226 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:51.898350 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6d4c44480ae94a3fafe4c6272b378009"
format_stamp: "Formatted at 2024-05-01 07:57:51 on dist-test-slave-qm51"
I20240501 07:57:51.898437 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:51.915803 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:51.916461 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:51.916625 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:51.917181 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:51.918067 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:51.918116 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:51.918176 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:51.918206 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:51.927722 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:45479
I20240501 07:57:51.927862 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:45479 every 8 connection(s)
I20240501 07:57:51.930223 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:57:51.935626 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 501
I20240501 07:57:51.935707 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:57:51.937964 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:57:51.939036 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44595
I20240501 07:57:51.939271 629 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:51.939754 629 heartbeater.cc:511] Master 127.0.99.190:44595 requested a full tablet report, sending...
I20240501 07:57:51.941282 440 ts_manager.cc:194] Registered new tserver with Master: 6d4c44480ae94a3fafe4c6272b378009 (127.0.99.129:45479)
I20240501 07:57:51.943084 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:59215
W20240501 07:57:52.042698 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:52.042932 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:52.042989 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:52.045877 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:52.045974 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:57:52.049394 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:52.050186 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:52.051247 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:52.059670 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.060567 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.061234 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:52.062006 632 server_base.cc:1034] running on GCE node
I20240501 07:57:52.062338 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:52.062873 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:52.064029 632 hybrid_clock.cc:648] HybridClock initialized: now 1714550272064027 us; error 43 us; skew 500 ppm
I20240501 07:57:52.065820 632 webserver.cc:460] Webserver started at http://127.0.99.130:45989/ using document root <none> and password file <none>
I20240501 07:57:52.066365 632 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:52.066416 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:52.066620 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:52.068271 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "53704ac0b6c9467a8e7d057dad1945a1"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.068725 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "53704ac0b6c9467a8e7d057dad1945a1"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.071902 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:57:52.073935 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.074867 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:52.074972 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "53704ac0b6c9467a8e7d057dad1945a1"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.075083 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:52.090849 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:52.091464 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:52.091611 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:52.092123 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:52.092969 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:52.093014 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.093050 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:52.093065 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.102605 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36191
I20240501 07:57:52.102634 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36191 every 8 connection(s)
I20240501 07:57:52.103618 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:57:52.111514 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44595
I20240501 07:57:52.111701 760 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:52.112165 760 heartbeater.cc:511] Master 127.0.99.190:44595 requested a full tablet report, sending...
I20240501 07:57:52.113062 440 ts_manager.cc:194] Registered new tserver with Master: 53704ac0b6c9467a8e7d057dad1945a1 (127.0.99.130:36191)
I20240501 07:57:52.113416 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 632
I20240501 07:57:52.113507 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:57:52.113672 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:60993
I20240501 07:57:52.115630 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 07:57:52.221989 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:52.222193 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:52.222234 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:52.225222 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:52.225319 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:57:52.228785 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:52.229576 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:52.230671 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:52.238910 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.239825 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.240496 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:52.241333 763 server_base.cc:1034] running on GCE node
I20240501 07:57:52.241703 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:52.242233 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:52.243420 763 hybrid_clock.cc:648] HybridClock initialized: now 1714550272243394 us; error 40 us; skew 500 ppm
I20240501 07:57:52.245247 763 webserver.cc:460] Webserver started at http://127.0.99.131:34985/ using document root <none> and password file <none>
I20240501 07:57:52.245796 763 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:52.245857 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:52.246024 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:52.247676 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5d4c37205d274ae7a4850ec19364e1e4"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.248183 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5d4c37205d274ae7a4850ec19364e1e4"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.251463 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:57:52.253554 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.254585 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:52.254683 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5d4c37205d274ae7a4850ec19364e1e4"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.254770 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:52.271947 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:52.272547 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:52.272701 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:52.273202 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:52.274057 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:52.274106 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.274165 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:52.274191 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.284119 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38497
I20240501 07:57:52.284148 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38497 every 8 connection(s)
I20240501 07:57:52.285058 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:57:52.290933 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 763
I20240501 07:57:52.291069 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:57:52.293208 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:57:52.293975 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44595
I20240501 07:57:52.294205 891 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:52.294871 891 heartbeater.cc:511] Master 127.0.99.190:44595 requested a full tablet report, sending...
I20240501 07:57:52.295940 440 ts_manager.cc:194] Registered new tserver with Master: 5d4c37205d274ae7a4850ec19364e1e4 (127.0.99.131:38497)
I20240501 07:57:52.296510 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:57157
W20240501 07:57:52.399663 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:52.399863 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:52.399910 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:52.403087 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:52.403196 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:57:52.406585 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:52.407451 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:52.408499 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:52.417199 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.418280 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.419317 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:52.420394 894 server_base.cc:1034] running on GCE node
I20240501 07:57:52.421119 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:52.421751 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:52.422904 894 hybrid_clock.cc:648] HybridClock initialized: now 1714550272422882 us; error 40 us; skew 500 ppm
I20240501 07:57:52.425125 894 webserver.cc:460] Webserver started at http://127.0.99.132:41411/ using document root <none> and password file <none>
I20240501 07:57:52.425752 894 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:52.425823 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:52.426057 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:52.428735 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "955bd65bfbd34993801db24fdb4c7126"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.429265 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "955bd65bfbd34993801db24fdb4c7126"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.432581 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:57:52.434830 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.435783 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:52.435886 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "955bd65bfbd34993801db24fdb4c7126"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.435963 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:52.449326 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:52.449879 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:52.450024 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:52.450510 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:52.451433 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:52.451498 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.451555 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:52.451588 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.461314 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34881
I20240501 07:57:52.461333 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34881 every 8 connection(s)
I20240501 07:57:52.462226 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:57:52.469019 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 894
I20240501 07:57:52.469135 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:57:52.471032 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44595
I20240501 07:57:52.471283 1022 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:52.471410 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--builtin_ntp_servers=127.0.99.148:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:57:52.471940 1022 heartbeater.cc:511] Master 127.0.99.190:44595 requested a full tablet report, sending...
I20240501 07:57:52.472904 440 ts_manager.cc:194] Registered new tserver with Master: 955bd65bfbd34993801db24fdb4c7126 (127.0.99.132:34881)
I20240501 07:57:52.473450 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:49097
W20240501 07:57:52.578052 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:52.578280 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:52.578332 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:52.581446 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:52.581550 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:57:52.585119 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44595
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:52.585991 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:52.587090 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:52.595547 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.596333 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:52.597116 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:52.597930 1025 server_base.cc:1034] running on GCE node
I20240501 07:57:52.598295 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:52.598835 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:52.600023 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714550272600007 us; error 32 us; skew 500 ppm
I20240501 07:57:52.601867 1025 webserver.cc:460] Webserver started at http://127.0.99.133:40001/ using document root <none> and password file <none>
I20240501 07:57:52.602450 1025 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:52.602521 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:52.602694 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:52.604355 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "29ca1ccd024446579529ed86b8db3bb3"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.604837 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "29ca1ccd024446579529ed86b8db3bb3"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.608197 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:57:52.610333 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.611464 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:57:52.611570 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "29ca1ccd024446579529ed86b8db3bb3"
format_stamp: "Formatted at 2024-05-01 07:57:52 on dist-test-slave-qm51"
I20240501 07:57:52.611655 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:52.622211 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:52.622896 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:52.623086 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:52.623623 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:52.624527 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:52.624580 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.624619 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:52.624651 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:52.633997 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44267
I20240501 07:57:52.634020 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44267 every 8 connection(s)
I20240501 07:57:52.634958 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:57:52.636874 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1025
I20240501 07:57:52.636983 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:57:52.644253 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44595
I20240501 07:57:52.644441 1153 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:52.644896 1153 heartbeater.cc:511] Master 127.0.99.190:44595 requested a full tablet report, sending...
I20240501 07:57:52.645776 440 ts_manager.cc:194] Registered new tserver with Master: 29ca1ccd024446579529ed86b8db3bb3 (127.0.99.133:44267)
I20240501 07:57:52.646299 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:47295
I20240501 07:57:52.651104 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:57:52.661747 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52706:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:57:52.679529 563 tablet_service.cc:1467] Processing CreateTablet for tablet 607116f105184d32a50b122606d072ea (DEFAULT_TABLE table=TestTable [id=3fb5993b55df41bda63eb61b1cdc8ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:52.679569 694 tablet_service.cc:1467] Processing CreateTablet for tablet 607116f105184d32a50b122606d072ea (DEFAULT_TABLE table=TestTable [id=3fb5993b55df41bda63eb61b1cdc8ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:52.679620 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 607116f105184d32a50b122606d072ea (DEFAULT_TABLE table=TestTable [id=3fb5993b55df41bda63eb61b1cdc8ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:52.680372 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607116f105184d32a50b122606d072ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:52.680414 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607116f105184d32a50b122606d072ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:52.680414 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607116f105184d32a50b122606d072ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:52.686568 1172 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Bootstrap starting.
I20240501 07:57:52.686576 1170 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Bootstrap starting.
I20240501 07:57:52.686997 1171 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Bootstrap starting.
I20240501 07:57:52.688475 1170 tablet_bootstrap.cc:654] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:52.688759 1172 tablet_bootstrap.cc:654] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:52.688759 1171 tablet_bootstrap.cc:654] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:52.689146 1170 log.cc:824] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:52.689474 1172 log.cc:824] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:52.689474 1171 log.cc:824] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:52.691232 1170 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: No bootstrap required, opened a new log
I20240501 07:57:52.691406 1170 ts_tablet_manager.cc:1367] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:57:52.691778 1172 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: No bootstrap required, opened a new log
I20240501 07:57:52.691232 1171 tablet_bootstrap.cc:492] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: No bootstrap required, opened a new log
I20240501 07:57:52.691990 1172 ts_tablet_manager.cc:1367] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 07:57:52.692183 1171 ts_tablet_manager.cc:1367] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 07:57:52.694695 1170 raft_consensus.cc:348] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.695034 1170 raft_consensus.cc:374] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:52.695137 1170 raft_consensus.cc:729] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4c44480ae94a3fafe4c6272b378009, State: Initialized, Role: FOLLOWER
I20240501 07:57:52.695633 1170 consensus_queue.cc:260] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.695825 1171 raft_consensus.cc:348] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.696023 1171 raft_consensus.cc:374] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:52.696113 1171 raft_consensus.cc:729] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ca1ccd024446579529ed86b8db3bb3, State: Initialized, Role: FOLLOWER
I20240501 07:57:52.696372 1172 raft_consensus.cc:348] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.696476 1171 consensus_queue.cc:260] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.696558 1172 raft_consensus.cc:374] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:52.696596 1170 ts_tablet_manager.cc:1398] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:57:52.696612 1172 raft_consensus.cc:729] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53704ac0b6c9467a8e7d057dad1945a1, State: Initialized, Role: FOLLOWER
I20240501 07:57:52.696954 629 heartbeater.cc:503] Master 127.0.99.190:44595 was elected leader, sending a full tablet report...
I20240501 07:57:52.697058 1153 heartbeater.cc:503] Master 127.0.99.190:44595 was elected leader, sending a full tablet report...
I20240501 07:57:52.697039 1172 consensus_queue.cc:260] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.697793 1172 ts_tablet_manager.cc:1398] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240501 07:57:52.698026 1171 ts_tablet_manager.cc:1398] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:57:52.698105 760 heartbeater.cc:503] Master 127.0.99.190:44595 was elected leader, sending a full tablet report...
I20240501 07:57:52.729285 1178 raft_consensus.cc:482] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:57:52.729499 1178 raft_consensus.cc:504] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.730564 1178 leader_election.cc:288] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ca1ccd024446579529ed86b8db3bb3 (127.0.99.133:44267), 6d4c44480ae94a3fafe4c6272b378009 (127.0.99.129:45479)
I20240501 07:57:52.734719 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607116f105184d32a50b122606d072ea" candidate_uuid: "53704ac0b6c9467a8e7d057dad1945a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4c44480ae94a3fafe4c6272b378009" is_pre_election: true
I20240501 07:57:52.735095 583 raft_consensus.cc:2454] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53704ac0b6c9467a8e7d057dad1945a1 in term 0.
I20240501 07:57:52.735105 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607116f105184d32a50b122606d072ea" candidate_uuid: "53704ac0b6c9467a8e7d057dad1945a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ca1ccd024446579529ed86b8db3bb3" is_pre_election: true
I20240501 07:57:52.735363 1107 raft_consensus.cc:2454] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53704ac0b6c9467a8e7d057dad1945a1 in term 0.
I20240501 07:57:52.735515 647 leader_election.cc:302] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53704ac0b6c9467a8e7d057dad1945a1, 6d4c44480ae94a3fafe4c6272b378009; no voters:
I20240501 07:57:52.735757 1178 raft_consensus.cc:2781] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:57:52.735843 1178 raft_consensus.cc:482] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:57:52.735895 1178 raft_consensus.cc:3037] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:52.736734 1178 raft_consensus.cc:504] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.737018 1178 leader_election.cc:288] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [CANDIDATE]: Term 1 election: Requested vote from peers 29ca1ccd024446579529ed86b8db3bb3 (127.0.99.133:44267), 6d4c44480ae94a3fafe4c6272b378009 (127.0.99.129:45479)
I20240501 07:57:52.737316 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607116f105184d32a50b122606d072ea" candidate_uuid: "53704ac0b6c9467a8e7d057dad1945a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ca1ccd024446579529ed86b8db3bb3"
I20240501 07:57:52.737354 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607116f105184d32a50b122606d072ea" candidate_uuid: "53704ac0b6c9467a8e7d057dad1945a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4c44480ae94a3fafe4c6272b378009"
I20240501 07:57:52.737458 1107 raft_consensus.cc:3037] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:52.737458 583 raft_consensus.cc:3037] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:52.738503 583 raft_consensus.cc:2454] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53704ac0b6c9467a8e7d057dad1945a1 in term 1.
I20240501 07:57:52.738503 1107 raft_consensus.cc:2454] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53704ac0b6c9467a8e7d057dad1945a1 in term 1.
I20240501 07:57:52.738797 647 leader_election.cc:302] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53704ac0b6c9467a8e7d057dad1945a1, 6d4c44480ae94a3fafe4c6272b378009; no voters:
I20240501 07:57:52.738945 1178 raft_consensus.cc:2781] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:52.739192 1178 raft_consensus.cc:686] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [term 1 LEADER]: Becoming Leader. State: Replica: 53704ac0b6c9467a8e7d057dad1945a1, State: Running, Role: LEADER
I20240501 07:57:52.739491 1178 consensus_queue.cc:237] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } }
I20240501 07:57:52.742309 440 catalog_manager.cc:5484] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53704ac0b6c9467a8e7d057dad1945a1 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "53704ac0b6c9467a8e7d057dad1945a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ca1ccd024446579529ed86b8db3bb3" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4c44480ae94a3fafe4c6272b378009" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53704ac0b6c9467a8e7d057dad1945a1" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36191 } health_report { overall_health: HEALTHY } } }
I20240501 07:57:52.755734 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:57:52.757273 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6d4c44480ae94a3fafe4c6272b378009 to finish bootstrapping
I20240501 07:57:52.762507 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 53704ac0b6c9467a8e7d057dad1945a1 to finish bootstrapping
I20240501 07:57:52.767083 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5d4c37205d274ae7a4850ec19364e1e4 to finish bootstrapping
I20240501 07:57:52.772624 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 955bd65bfbd34993801db24fdb4c7126 to finish bootstrapping
I20240501 07:57:52.777549 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 29ca1ccd024446579529ed86b8db3bb3 to finish bootstrapping
W20240501 07:57:52.854887 761 tablet.cc:2369] T 607116f105184d32a50b122606d072ea P 53704ac0b6c9467a8e7d057dad1945a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:57:52.885911 1154 tablet.cc:2369] T 607116f105184d32a50b122606d072ea P 29ca1ccd024446579529ed86b8db3bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:57:52.931612 630 tablet.cc:2369] T 607116f105184d32a50b122606d072ea P 6d4c44480ae94a3fafe4c6272b378009: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:57:52.962754 694 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:57:52.962831 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:52.964637 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:52.966617 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:52.967248 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:53.055269 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:53.056248 694 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:57:53.057106 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:53.058311 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:53.063208 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:53.095901 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 501
I20240501 07:57:53.105093 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 632
I20240501 07:57:53.110808 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 763
I20240501 07:57:53.116720 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 894
I20240501 07:57:53.122076 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1025
I20240501 07:57:53.127823 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 410
2024-05-01T07:57:53Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1646 ms)
[----------] 1 test from AdminCliTest (1646 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 07:57:53.150347 398 test_util.cc:268] Using random seed: -671412858
I20240501 07:57:53.151242 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:57:53.151290 398 ts_itest-base.cc:113] --------------
I20240501 07:57:53.151302 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:57:53.151314 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:57:53.151325 398 ts_itest-base.cc:116] --------------
2024-05-01T07:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:57:53Z Disabled control of system clock
I20240501 07:57:53.158756 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36441
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36441
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 07:57:53.261090 1259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:53.261336 1259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:53.261406 1259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:53.264380 1259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:53.264448 1259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:57:53.264484 1259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:53.264525 1259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:57:53.264561 1259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:57:53.268083 1259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36441
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:53.268990 1259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:53.270061 1259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:53.277935 1265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.278676 1267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.279515 1264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:53.280372 1259 server_base.cc:1034] running on GCE node
I20240501 07:57:53.280900 1259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:53.281855 1259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:53.283027 1259 hybrid_clock.cc:648] HybridClock initialized: now 1714550273282996 us; error 30 us; skew 500 ppm
I20240501 07:57:53.284940 1259 webserver.cc:460] Webserver started at http://127.0.99.190:42729/ using document root <none> and password file <none>
I20240501 07:57:53.285486 1259 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:53.285547 1259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:53.285715 1259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:53.287387 1259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b6c8402e15644fe4a8d7df60a380dd9c"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.287843 1259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b6c8402e15644fe4a8d7df60a380dd9c"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.291263 1259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:57:53.293282 1273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.294207 1259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:53.294320 1259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b6c8402e15644fe4a8d7df60a380dd9c"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.294397 1259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:53.322638 1259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:53.323287 1259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:53.323439 1259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:53.329830 1259 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36441
I20240501 07:57:53.329844 1325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36441 every 8 connection(s)
I20240501 07:57:53.330760 1259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:57:53.333241 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:53.333281 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1259
I20240501 07:57:53.333384 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:57:53.338814 1326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: Bootstrap starting.
I20240501 07:57:53.340994 1326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:53.341684 1326 log.cc:824] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:53.343231 1326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: No bootstrap required, opened a new log
I20240501 07:57:53.345641 1326 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } }
I20240501 07:57:53.345803 1326 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:53.345865 1326 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c8402e15644fe4a8d7df60a380dd9c, State: Initialized, Role: FOLLOWER
I20240501 07:57:53.346261 1326 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } }
I20240501 07:57:53.346377 1326 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:57:53.346436 1326 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:57:53.346678 1326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:53.347450 1326 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } }
I20240501 07:57:53.347779 1326 leader_election.cc:302] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c8402e15644fe4a8d7df60a380dd9c; no voters:
I20240501 07:57:53.348023 1326 leader_election.cc:288] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:57:53.348114 1331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:53.348347 1331 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [term 1 LEADER]: Becoming Leader. State: Replica: b6c8402e15644fe4a8d7df60a380dd9c, State: Running, Role: LEADER
I20240501 07:57:53.349066 1326 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:57:53.349037 1331 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } }
I20240501 07:57:53.350440 1332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } } }
I20240501 07:57:53.350488 1333 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c8402e15644fe4a8d7df60a380dd9c. Latest consensus state: current_term: 1 leader_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c8402e15644fe4a8d7df60a380dd9c" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36441 } } }
I20240501 07:57:53.350562 1332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [sys.catalog]: This master's current role is: LEADER
I20240501 07:57:53.350589 1333 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c [sys.catalog]: This master's current role is: LEADER
W20240501 07:57:53.353536 1346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:57:53.353598 1346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:57:53.353674 1342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:57:53.354331 1342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:57:53.358505 1342 catalog_manager.cc:1347] Generated new cluster ID: 9bd329b886de468c99b94d377cab1681
I20240501 07:57:53.358562 1342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:57:53.364324 1342 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:57:53.365113 1342 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:57:53.370272 1342 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b6c8402e15644fe4a8d7df60a380dd9c: Generated new TSK 0
I20240501 07:57:53.370949 1342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:57:53.380118 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:57:53.498715 1350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:53.498986 1350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:53.499095 1350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:53.502143 1350 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:53.502207 1350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:53.502298 1350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:57:53.505985 1350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--never_fsync=true
--heap_profile_path=/tmp/kudu.1350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:53.506865 1350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:53.508020 1350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:53.517742 1355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.518513 1358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.519309 1356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:53.520138 1350 server_base.cc:1034] running on GCE node
I20240501 07:57:53.520502 1350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:53.521065 1350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:53.522226 1350 hybrid_clock.cc:648] HybridClock initialized: now 1714550273522210 us; error 36 us; skew 500 ppm
I20240501 07:57:53.524158 1350 webserver.cc:460] Webserver started at http://127.0.99.129:42379/ using document root <none> and password file <none>
I20240501 07:57:53.524704 1350 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:53.524765 1350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:53.524937 1350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:53.526623 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d2b854e248f148edb743f28b2df770be"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.527141 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d2b854e248f148edb743f28b2df770be"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.530364 1350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:57:53.532466 1364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.533391 1350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:57:53.533493 1350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d2b854e248f148edb743f28b2df770be"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.533572 1350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:53.565825 1350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:53.566447 1350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:53.566606 1350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:53.567159 1350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:53.568033 1350 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:53.568081 1350 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.568141 1350 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:53.568171 1350 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.576347 1350 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38623
I20240501 07:57:53.576382 1477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38623 every 8 connection(s)
I20240501 07:57:53.577251 1350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:57:53.585705 1478 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36441
I20240501 07:57:53.585907 1478 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:53.586287 1478 heartbeater.cc:511] Master 127.0.99.190:36441 requested a full tablet report, sending...
I20240501 07:57:53.586405 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1350
I20240501 07:57:53.586508 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:57:53.587666 1290 ts_manager.cc:194] Registered new tserver with Master: d2b854e248f148edb743f28b2df770be (127.0.99.129:38623)
I20240501 07:57:53.588521 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:53.588876 1290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48093
W20240501 07:57:53.692646 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:53.692870 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:53.692950 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:53.695952 1481 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:53.696049 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:53.696141 1481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:57:53.699690 1481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--never_fsync=true
--heap_profile_path=/tmp/kudu.1481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:53.700510 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:53.701556 1481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:53.710145 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.711043 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.711764 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:53.712678 1481 server_base.cc:1034] running on GCE node
I20240501 07:57:53.713096 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:53.713694 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:53.714838 1481 hybrid_clock.cc:648] HybridClock initialized: now 1714550273714818 us; error 35 us; skew 500 ppm
I20240501 07:57:53.716697 1481 webserver.cc:460] Webserver started at http://127.0.99.130:36101/ using document root <none> and password file <none>
I20240501 07:57:53.717218 1481 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:53.717273 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:53.717449 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:53.719082 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "29f12b0c1c5e453e97e78fb863c1cd5e"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.719534 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "29f12b0c1c5e453e97e78fb863c1cd5e"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.722784 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 07:57:53.724850 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.725963 1481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:57:53.726069 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "29f12b0c1c5e453e97e78fb863c1cd5e"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.726171 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:53.746397 1481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:53.747004 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:53.747176 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:53.747691 1481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:53.748548 1481 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:53.748623 1481 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.748679 1481 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:53.748713 1481 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.756920 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33645
I20240501 07:57:53.756964 1608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33645 every 8 connection(s)
I20240501 07:57:53.757869 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:57:53.763698 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1481
I20240501 07:57:53.763809 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:57:53.766109 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:53.766378 1609 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36441
I20240501 07:57:53.766600 1609 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:53.767109 1609 heartbeater.cc:511] Master 127.0.99.190:36441 requested a full tablet report, sending...
I20240501 07:57:53.768131 1290 ts_manager.cc:194] Registered new tserver with Master: 29f12b0c1c5e453e97e78fb863c1cd5e (127.0.99.130:33645)
I20240501 07:57:53.768692 1290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:53189
W20240501 07:57:53.868863 1612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:53.869067 1612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:53.869113 1612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:53.872074 1612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:53.872130 1612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:53.872213 1612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:57:53.875669 1612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--never_fsync=true
--heap_profile_path=/tmp/kudu.1612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:53.876457 1612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:53.877491 1612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:53.885941 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.886749 1617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:53.887579 1620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:53.888407 1612 server_base.cc:1034] running on GCE node
I20240501 07:57:53.888778 1612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:53.889324 1612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:53.890503 1612 hybrid_clock.cc:648] HybridClock initialized: now 1714550273890486 us; error 30 us; skew 500 ppm
I20240501 07:57:53.892493 1612 webserver.cc:460] Webserver started at http://127.0.99.131:36369/ using document root <none> and password file <none>
I20240501 07:57:53.893049 1612 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:53.893116 1612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:53.893304 1612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:53.894942 1612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "49c46349b22d469d8be3d8e4ba36480a"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.895457 1612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "49c46349b22d469d8be3d8e4ba36480a"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.898701 1612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:57:53.900930 1626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.901912 1612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:53.902010 1612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "49c46349b22d469d8be3d8e4ba36480a"
format_stamp: "Formatted at 2024-05-01 07:57:53 on dist-test-slave-qm51"
I20240501 07:57:53.902109 1612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:53.913754 1612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:53.914327 1612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:53.914474 1612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:53.914976 1612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:53.915839 1612 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:53.915885 1612 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.915923 1612 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:53.915944 1612 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:53.923864 1612 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40203
I20240501 07:57:53.923918 1739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40203 every 8 connection(s)
I20240501 07:57:53.924736 1612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:57:53.931564 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1612
I20240501 07:57:53.931671 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:57:53.933307 1740 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36441
I20240501 07:57:53.933501 1740 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:53.933650 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:53.933962 1740 heartbeater.cc:511] Master 127.0.99.190:36441 requested a full tablet report, sending...
I20240501 07:57:53.934885 1290 ts_manager.cc:194] Registered new tserver with Master: 49c46349b22d469d8be3d8e4ba36480a (127.0.99.131:40203)
I20240501 07:57:53.935633 1290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:51359
W20240501 07:57:54.036717 1743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:54.036939 1743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:54.036980 1743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:54.039888 1743 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:54.039942 1743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:54.040006 1743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:57:54.043430 1743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--never_fsync=true
--heap_profile_path=/tmp/kudu.1743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:54.044226 1743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:54.045293 1743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:54.054054 1749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:54.054872 1751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:54.055663 1748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:54.056479 1743 server_base.cc:1034] running on GCE node
I20240501 07:57:54.056864 1743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:54.057395 1743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:54.058540 1743 hybrid_clock.cc:648] HybridClock initialized: now 1714550274058537 us; error 33 us; skew 500 ppm
I20240501 07:57:54.060493 1743 webserver.cc:460] Webserver started at http://127.0.99.132:42951/ using document root <none> and password file <none>
I20240501 07:57:54.061039 1743 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:54.061100 1743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:54.061268 1743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:54.062875 1743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8ad4baec6ff44c719f822d5a7ace0c63"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.063357 1743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8ad4baec6ff44c719f822d5a7ace0c63"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.066507 1743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:57:54.068742 1757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.069715 1743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 07:57:54.069801 1743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8ad4baec6ff44c719f822d5a7ace0c63"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.069869 1743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:54.093770 1743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:54.094285 1743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:54.094410 1743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:54.094859 1743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:54.095680 1743 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:54.095724 1743 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.095762 1743 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:54.095782 1743 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.104017 1743 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34237
I20240501 07:57:54.104053 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34237 every 8 connection(s)
I20240501 07:57:54.104885 1743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:57:54.109519 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1743
I20240501 07:57:54.109771 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:57:54.111994 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--builtin_ntp_servers=127.0.99.148:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:54.113865 1871 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36441
I20240501 07:57:54.114078 1871 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:54.114591 1871 heartbeater.cc:511] Master 127.0.99.190:36441 requested a full tablet report, sending...
I20240501 07:57:54.115605 1290 ts_manager.cc:194] Registered new tserver with Master: 8ad4baec6ff44c719f822d5a7ace0c63 (127.0.99.132:34237)
I20240501 07:57:54.116207 1290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55625
W20240501 07:57:54.216740 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:54.216964 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:54.217033 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:54.220101 1874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:54.220172 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:54.220264 1874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:57:54.223750 1874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36441
--never_fsync=true
--heap_profile_path=/tmp/kudu.1874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:54.224637 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:54.225718 1874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:54.234776 1880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:54.235476 1879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:54.236291 1882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:54.237064 1874 server_base.cc:1034] running on GCE node
I20240501 07:57:54.237408 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:54.237972 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:54.239081 1874 hybrid_clock.cc:648] HybridClock initialized: now 1714550274239060 us; error 40 us; skew 500 ppm
I20240501 07:57:54.240976 1874 webserver.cc:460] Webserver started at http://127.0.99.133:35279/ using document root <none> and password file <none>
I20240501 07:57:54.241482 1874 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:54.241536 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:54.241712 1874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:54.243356 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "49f213b54d024c1fb6a71fd045e77ea7"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.243808 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "49f213b54d024c1fb6a71fd045e77ea7"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.246953 1874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:57:54.249096 1888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.250142 1874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:57:54.250252 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "49f213b54d024c1fb6a71fd045e77ea7"
format_stamp: "Formatted at 2024-05-01 07:57:54 on dist-test-slave-qm51"
I20240501 07:57:54.250340 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:54.276357 1874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:54.276906 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:54.277030 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:54.277475 1874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:54.278285 1874 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:54.278331 1874 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.278370 1874 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:54.278391 1874 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:54.286432 1874 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41447
I20240501 07:57:54.286458 2001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41447 every 8 connection(s)
I20240501 07:57:54.287334 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:57:54.288110 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 1874
I20240501 07:57:54.288187 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:57:54.295809 2002 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36441
I20240501 07:57:54.296021 2002 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:54.296416 2002 heartbeater.cc:511] Master 127.0.99.190:36441 requested a full tablet report, sending...
I20240501 07:57:54.297258 1290 ts_manager.cc:194] Registered new tserver with Master: 49f213b54d024c1fb6a71fd045e77ea7 (127.0.99.133:41447)
I20240501 07:57:54.297776 1290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57659
I20240501 07:57:54.301751 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:57:54.310070 1290 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56782:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:57:54.327294 1936 tablet_service.cc:1467] Processing CreateTablet for tablet fa07af5d6f62436bbbb6ffd7845f2534 (DEFAULT_TABLE table=TestTable [id=2e565444a816440b97e3fca1fc3dd20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:54.327294 1412 tablet_service.cc:1467] Processing CreateTablet for tablet fa07af5d6f62436bbbb6ffd7845f2534 (DEFAULT_TABLE table=TestTable [id=2e565444a816440b97e3fca1fc3dd20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:54.327296 1543 tablet_service.cc:1467] Processing CreateTablet for tablet fa07af5d6f62436bbbb6ffd7845f2534 (DEFAULT_TABLE table=TestTable [id=2e565444a816440b97e3fca1fc3dd20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:54.328161 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa07af5d6f62436bbbb6ffd7845f2534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:54.328173 1543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa07af5d6f62436bbbb6ffd7845f2534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:54.328173 1412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa07af5d6f62436bbbb6ffd7845f2534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:54.334510 2019 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Bootstrap starting.
I20240501 07:57:54.335273 2021 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Bootstrap starting.
I20240501 07:57:54.336629 2019 tablet_bootstrap.cc:654] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:54.336925 2021 tablet_bootstrap.cc:654] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:54.337400 2019 log.cc:824] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:54.337538 2021 log.cc:824] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:54.338040 2020 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: Bootstrap starting.
I20240501 07:57:54.339342 2019 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: No bootstrap required, opened a new log
I20240501 07:57:54.339569 2019 ts_tablet_manager.cc:1367] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 07:57:54.339844 2021 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: No bootstrap required, opened a new log
I20240501 07:57:54.339998 2021 ts_tablet_manager.cc:1367] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:57:54.340001 2020 tablet_bootstrap.cc:654] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:54.340683 2020 log.cc:824] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:54.342190 2020 tablet_bootstrap.cc:492] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: No bootstrap required, opened a new log
I20240501 07:57:54.342334 2020 ts_tablet_manager.cc:1367] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:57:54.343711 2019 raft_consensus.cc:348] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.344040 2019 raft_consensus.cc:374] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:54.344158 2019 raft_consensus.cc:729] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f213b54d024c1fb6a71fd045e77ea7, State: Initialized, Role: FOLLOWER
I20240501 07:57:54.344422 2021 raft_consensus.cc:348] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.344627 2021 raft_consensus.cc:374] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:54.344684 2021 raft_consensus.cc:729] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f12b0c1c5e453e97e78fb863c1cd5e, State: Initialized, Role: FOLLOWER
I20240501 07:57:54.344857 2019 consensus_queue.cc:260] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.345091 2021 consensus_queue.cc:260] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.345631 2002 heartbeater.cc:503] Master 127.0.99.190:36441 was elected leader, sending a full tablet report...
I20240501 07:57:54.345870 2021 ts_tablet_manager.cc:1398] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 07:57:54.345870 2019 ts_tablet_manager.cc:1398] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240501 07:57:54.345758 2020 raft_consensus.cc:348] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.345952 2020 raft_consensus.cc:374] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:54.346009 2020 raft_consensus.cc:729] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b854e248f148edb743f28b2df770be, State: Initialized, Role: FOLLOWER
I20240501 07:57:54.346180 1609 heartbeater.cc:503] Master 127.0.99.190:36441 was elected leader, sending a full tablet report...
I20240501 07:57:54.346354 2020 consensus_queue.cc:260] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.346908 1478 heartbeater.cc:503] Master 127.0.99.190:36441 was elected leader, sending a full tablet report...
I20240501 07:57:54.347155 2020 ts_tablet_manager.cc:1398] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:57:54.478677 2027 raft_consensus.cc:482] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:57:54.478886 2027 raft_consensus.cc:504] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.480060 2027 leader_election.cc:288] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f12b0c1c5e453e97e78fb863c1cd5e (127.0.99.130:33645), 49f213b54d024c1fb6a71fd045e77ea7 (127.0.99.133:41447)
I20240501 07:57:54.484855 1563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa07af5d6f62436bbbb6ffd7845f2534" candidate_uuid: "d2b854e248f148edb743f28b2df770be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" is_pre_election: true
I20240501 07:57:54.484853 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa07af5d6f62436bbbb6ffd7845f2534" candidate_uuid: "d2b854e248f148edb743f28b2df770be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f213b54d024c1fb6a71fd045e77ea7" is_pre_election: true
I20240501 07:57:54.485107 1563 raft_consensus.cc:2454] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2b854e248f148edb743f28b2df770be in term 0.
I20240501 07:57:54.485121 1956 raft_consensus.cc:2454] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2b854e248f148edb743f28b2df770be in term 0.
I20240501 07:57:54.485502 1367 leader_election.cc:302] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f12b0c1c5e453e97e78fb863c1cd5e, d2b854e248f148edb743f28b2df770be; no voters:
I20240501 07:57:54.485790 2027 raft_consensus.cc:2781] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:57:54.485855 2027 raft_consensus.cc:482] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:57:54.485903 2027 raft_consensus.cc:3037] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:54.486738 2027 raft_consensus.cc:504] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.487094 2027 leader_election.cc:288] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [CANDIDATE]: Term 1 election: Requested vote from peers 29f12b0c1c5e453e97e78fb863c1cd5e (127.0.99.130:33645), 49f213b54d024c1fb6a71fd045e77ea7 (127.0.99.133:41447)
I20240501 07:57:54.487452 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa07af5d6f62436bbbb6ffd7845f2534" candidate_uuid: "d2b854e248f148edb743f28b2df770be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f213b54d024c1fb6a71fd045e77ea7"
I20240501 07:57:54.487460 1563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa07af5d6f62436bbbb6ffd7845f2534" candidate_uuid: "d2b854e248f148edb743f28b2df770be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e"
I20240501 07:57:54.487591 1563 raft_consensus.cc:3037] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:54.487591 1956 raft_consensus.cc:3037] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:54.488732 1563 raft_consensus.cc:2454] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b854e248f148edb743f28b2df770be in term 1.
I20240501 07:57:54.488732 1956 raft_consensus.cc:2454] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2b854e248f148edb743f28b2df770be in term 1.
I20240501 07:57:54.489038 1368 leader_election.cc:302] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f213b54d024c1fb6a71fd045e77ea7, d2b854e248f148edb743f28b2df770be; no voters:
I20240501 07:57:54.489213 2027 raft_consensus.cc:2781] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:54.489423 2027 raft_consensus.cc:686] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [term 1 LEADER]: Becoming Leader. State: Replica: d2b854e248f148edb743f28b2df770be, State: Running, Role: LEADER
I20240501 07:57:54.489719 2027 consensus_queue.cc:237] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } }
I20240501 07:57:54.492444 1288 catalog_manager.cc:5484] T fa07af5d6f62436bbbb6ffd7845f2534 P d2b854e248f148edb743f28b2df770be reported cstate change: term changed from 0 to 1, leader changed from <none> to d2b854e248f148edb743f28b2df770be (127.0.99.129). New cstate: current_term: 1 leader_uuid: "d2b854e248f148edb743f28b2df770be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b854e248f148edb743f28b2df770be" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38623 } } peers { permanent_uuid: "29f12b0c1c5e453e97e78fb863c1cd5e" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33645 } } peers { permanent_uuid: "49f213b54d024c1fb6a71fd045e77ea7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41447 } } }
W20240501 07:57:54.509274 1610 tablet.cc:2369] T fa07af5d6f62436bbbb6ffd7845f2534 P 29f12b0c1c5e453e97e78fb863c1cd5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:57:54.510798 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:57:54.512161 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d2b854e248f148edb743f28b2df770be to finish bootstrapping
I20240501 07:57:54.517357 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 29f12b0c1c5e453e97e78fb863c1cd5e to finish bootstrapping
I20240501 07:57:54.521643 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 49c46349b22d469d8be3d8e4ba36480a to finish bootstrapping
I20240501 07:57:54.526744 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8ad4baec6ff44c719f822d5a7ace0c63 to finish bootstrapping
I20240501 07:57:54.531733 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 49f213b54d024c1fb6a71fd045e77ea7 to finish bootstrapping
I20240501 07:57:54.536032 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1350
W20240501 07:57:54.538111 2003 tablet.cc:2369] T fa07af5d6f62436bbbb6ffd7845f2534 P 49f213b54d024c1fb6a71fd045e77ea7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:57:54.720407 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.720505 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.720767 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.721319 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.797511 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.798158 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.798197 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.799124 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.891886 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.892045 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.892563 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.892849 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:54.937916 1740 heartbeater.cc:503] Master 127.0.99.190:36441 was elected leader, sending a full tablet report...
I20240501 07:57:55.104450 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.104581 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.105962 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.106781 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.121716 1871 heartbeater.cc:503] Master 127.0.99.190:36441 was elected leader, sending a full tablet report...
I20240501 07:57:55.178364 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.178869 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.179313 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.187238 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.289955 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.290418 1543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.291438 1936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.298322 1674 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:57:55.330682 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1481
I20240501 07:57:55.341135 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1612
I20240501 07:57:55.347687 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1743
I20240501 07:57:55.354301 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1874
I20240501 07:57:55.359918 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 1259
2024-05-01T07:57:55Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2233 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2233 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 07:57:55.383603 398 test_util.cc:268] Using random seed: -669179603
I20240501 07:57:55.384357 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:57:55.384394 398 ts_itest-base.cc:113] --------------
I20240501 07:57:55.384413 398 ts_itest-base.cc:114] 7 tablet servers
I20240501 07:57:55.384438 398 ts_itest-base.cc:115] 1 replicas per TS
I20240501 07:57:55.384464 398 ts_itest-base.cc:116] --------------
2024-05-01T07:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:57:55Z Disabled control of system clock
I20240501 07:57:55.392716 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33335
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:33335
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 07:57:55.504035 2217 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 07:57:55.504285 2217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:55.504313 2217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:55.504350 2217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:55.508157 2217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:55.508237 2217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:57:55.508261 2217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:55.508280 2217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:57:55.508296 2217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:57:55.511993 2217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:33335
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:55.512929 2217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:55.514078 2217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:55.521958 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.522789 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.523499 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:55.524403 2217 server_base.cc:1034] running on GCE node
I20240501 07:57:55.524869 2217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:55.525751 2217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:55.526898 2217 hybrid_clock.cc:648] HybridClock initialized: now 1714550275526885 us; error 29 us; skew 500 ppm
I20240501 07:57:55.530185 2217 webserver.cc:460] Webserver started at http://127.0.99.190:44231/ using document root <none> and password file <none>
I20240501 07:57:55.530774 2217 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:55.530999 2217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:55.531265 2217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:55.533032 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d8d12c170da7423b82335f6f72d0a66d"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.533509 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d8d12c170da7423b82335f6f72d0a66d"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.536926 2217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:57:55.539256 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.540298 2217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:57:55.540417 2217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d8d12c170da7423b82335f6f72d0a66d"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.540508 2217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:55.558519 2217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:55.559276 2217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:55.559455 2217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:55.566635 2217 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:33335
I20240501 07:57:55.566681 2283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:33335 every 8 connection(s)
I20240501 07:57:55.567700 2217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:57:55.569443 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2217
I20240501 07:57:55.569562 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:57:55.571482 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:55.576270 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: Bootstrap starting.
I20240501 07:57:55.578564 2284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:55.579461 2284 log.cc:824] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:55.581018 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: No bootstrap required, opened a new log
I20240501 07:57:55.583762 2284 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } }
I20240501 07:57:55.583958 2284 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:55.583995 2284 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d12c170da7423b82335f6f72d0a66d, State: Initialized, Role: FOLLOWER
I20240501 07:57:55.584386 2284 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } }
I20240501 07:57:55.584501 2284 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:57:55.584543 2284 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:57:55.584650 2284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:55.586719 2284 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } }
I20240501 07:57:55.587070 2284 leader_election.cc:302] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d12c170da7423b82335f6f72d0a66d; no voters:
I20240501 07:57:55.587364 2284 leader_election.cc:288] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:57:55.587502 2289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:55.587779 2289 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [term 1 LEADER]: Becoming Leader. State: Replica: d8d12c170da7423b82335f6f72d0a66d, State: Running, Role: LEADER
I20240501 07:57:55.588173 2289 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } }
I20240501 07:57:55.588503 2284 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:57:55.589821 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d12c170da7423b82335f6f72d0a66d. Latest consensus state: current_term: 1 leader_uuid: "d8d12c170da7423b82335f6f72d0a66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } } }
I20240501 07:57:55.589936 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [sys.catalog]: This master's current role is: LEADER
I20240501 07:57:55.590216 2290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d12c170da7423b82335f6f72d0a66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d12c170da7423b82335f6f72d0a66d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33335 } } }
I20240501 07:57:55.590310 2290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d [sys.catalog]: This master's current role is: LEADER
W20240501 07:57:55.593282 2304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:57:55.593358 2304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:57:55.593477 2299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:57:55.594280 2299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:57:55.599251 2299 catalog_manager.cc:1347] Generated new cluster ID: 7b230db298174a69bc3a48b7f490889f
I20240501 07:57:55.599342 2299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:57:55.610648 2299 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:57:55.611891 2299 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:57:55.618676 2299 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d8d12c170da7423b82335f6f72d0a66d: Generated new TSK 0
I20240501 07:57:55.619550 2299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:57:55.627892 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:57:55.737524 2308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:55.737738 2308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:55.737782 2308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:55.740892 2308 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:55.740963 2308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:55.741047 2308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:57:55.744908 2308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:55.745833 2308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:55.746979 2308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:55.755685 2314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.756484 2316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.757521 2313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:55.758349 2308 server_base.cc:1034] running on GCE node
I20240501 07:57:55.758711 2308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:55.759282 2308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:55.760502 2308 hybrid_clock.cc:648] HybridClock initialized: now 1714550275760491 us; error 27 us; skew 500 ppm
I20240501 07:57:55.762481 2308 webserver.cc:460] Webserver started at http://127.0.99.129:37019/ using document root <none> and password file <none>
I20240501 07:57:55.763083 2308 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:55.763149 2308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:55.763342 2308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:55.764963 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.765413 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.769456 2308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 07:57:55.771728 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.772763 2308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:57:55.772874 2308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.772969 2308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:55.802738 2308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:55.803480 2308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:55.803645 2308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:55.804184 2308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:55.805068 2308 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:55.805119 2308 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.805174 2308 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:55.805207 2308 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.813766 2308 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:45303
I20240501 07:57:55.813804 2435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:45303 every 8 connection(s)
I20240501 07:57:55.814693 2308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:57:55.823045 2436 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:55.823271 2436 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:55.823755 2436 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:55.823886 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2308
I20240501 07:57:55.824018 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:57:55.825103 2248 ts_manager.cc:194] Registered new tserver with Master: bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:57:55.826054 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:55.826331 2248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47101
W20240501 07:57:55.933210 2439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:55.933418 2439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:55.933468 2439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:55.936415 2439 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:55.936476 2439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:55.936540 2439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:57:55.940045 2439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:55.940939 2439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:55.942047 2439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:55.950747 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.951556 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:55.952392 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:55.953547 2439 server_base.cc:1034] running on GCE node
I20240501 07:57:55.953934 2439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:55.954492 2439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:55.955662 2439 hybrid_clock.cc:648] HybridClock initialized: now 1714550275955637 us; error 41 us; skew 500 ppm
I20240501 07:57:55.957628 2439 webserver.cc:460] Webserver started at http://127.0.99.130:45951/ using document root <none> and password file <none>
I20240501 07:57:55.958169 2439 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:55.958247 2439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:55.958420 2439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:55.960124 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.960614 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.963914 2439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:57:55.966053 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.967104 2439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:57:55.967203 2439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
format_stamp: "Formatted at 2024-05-01 07:57:55 on dist-test-slave-qm51"
I20240501 07:57:55.967286 2439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:55.977771 2439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:55.978332 2439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:55.978492 2439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:55.979156 2439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:55.980077 2439 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:55.980124 2439 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.980177 2439 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:55.980211 2439 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:55.988798 2439 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38583
I20240501 07:57:55.988817 2566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38583 every 8 connection(s)
I20240501 07:57:55.989820 2439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:57:55.991570 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2439
I20240501 07:57:55.991689 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:57:55.994890 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:56.001010 2567 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:56.001274 2567 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:56.001901 2567 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:56.003077 2248 ts_manager.cc:194] Registered new tserver with Master: fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:57:56.003624 2248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:46247
W20240501 07:57:56.107067 2570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:56.107301 2570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:56.107373 2570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:56.110620 2570 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:56.110721 2570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:56.110822 2570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:57:56.114465 2570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:56.115406 2570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:56.116559 2570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:56.125643 2578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.126441 2575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.127261 2576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.128170 2570 server_base.cc:1034] running on GCE node
I20240501 07:57:56.128556 2570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:56.129128 2570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:56.130282 2570 hybrid_clock.cc:648] HybridClock initialized: now 1714550276130270 us; error 26 us; skew 500 ppm
I20240501 07:57:56.132182 2570 webserver.cc:460] Webserver started at http://127.0.99.131:34307/ using document root <none> and password file <none>
I20240501 07:57:56.132742 2570 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:56.132794 2570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:56.132961 2570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:56.134589 2570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fe0605ab798147669ec83b685fc7ace0"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.135082 2570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fe0605ab798147669ec83b685fc7ace0"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.138425 2570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:57:56.140628 2584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.141623 2570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:57:56.141726 2570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fe0605ab798147669ec83b685fc7ace0"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.141798 2570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:56.153087 2570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:56.153649 2570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:56.153782 2570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:56.154230 2570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:56.155092 2570 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:56.155139 2570 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.155177 2570 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:56.155192 2570 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.163322 2570 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39669
I20240501 07:57:56.163370 2697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39669 every 8 connection(s)
I20240501 07:57:56.164234 2570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:57:56.170616 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2570
I20240501 07:57:56.170732 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:57:56.173218 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:56.173440 2698 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:56.173682 2698 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:56.174120 2698 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:56.175034 2248 ts_manager.cc:194] Registered new tserver with Master: fe0605ab798147669ec83b685fc7ace0 (127.0.99.131:39669)
I20240501 07:57:56.175606 2248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49745
W20240501 07:57:56.284027 2701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:56.284268 2701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:56.284340 2701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:56.287449 2701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:56.287537 2701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:56.287636 2701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:57:56.291429 2701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:56.292424 2701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:56.293922 2701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:56.304069 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.304944 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.305908 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.306871 2701 server_base.cc:1034] running on GCE node
I20240501 07:57:56.307372 2701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:56.308105 2701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:56.309283 2701 hybrid_clock.cc:648] HybridClock initialized: now 1714550276309262 us; error 44 us; skew 500 ppm
I20240501 07:57:56.311517 2701 webserver.cc:460] Webserver started at http://127.0.99.132:39639/ using document root <none> and password file <none>
I20240501 07:57:56.312117 2701 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:56.312177 2701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:56.312356 2701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:56.314270 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3199fb0a9d4d448687cd049ae66601c9"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.314814 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3199fb0a9d4d448687cd049ae66601c9"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.318378 2701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:57:56.320812 2715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.321904 2701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:57:56.322018 2701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3199fb0a9d4d448687cd049ae66601c9"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.322119 2701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:56.337502 2701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:56.338198 2701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:56.338366 2701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:56.338944 2701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:56.339900 2701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:56.339948 2701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.340010 2701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:56.340036 2701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.349752 2701 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35435
I20240501 07:57:56.349792 2828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35435 every 8 connection(s)
I20240501 07:57:56.350842 2701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:57:56.359535 2829 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:56.359794 2829 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:56.359948 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2701
I20240501 07:57:56.360067 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:57:56.360360 2829 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:56.361284 2247 ts_manager.cc:194] Registered new tserver with Master: 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132:35435)
I20240501 07:57:56.361857 2247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60245
I20240501 07:57:56.362214 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:57:56.480845 2832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:56.481067 2832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:56.481139 2832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:56.485383 2832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:56.485507 2832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:56.485638 2832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:57:56.489704 2832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:56.490662 2832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:56.491835 2832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:56.501271 2840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.502050 2837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.502925 2838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.503804 2832 server_base.cc:1034] running on GCE node
I20240501 07:57:56.504335 2832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:56.504951 2832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:56.506103 2832 hybrid_clock.cc:648] HybridClock initialized: now 1714550276506091 us; error 31 us; skew 500 ppm
I20240501 07:57:56.508054 2832 webserver.cc:460] Webserver started at http://127.0.99.133:39913/ using document root <none> and password file <none>
I20240501 07:57:56.508606 2832 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:56.508667 2832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:56.508841 2832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:56.510491 2832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f25b7c34fac14ab3bfe3c58386fcff04"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.510952 2832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f25b7c34fac14ab3bfe3c58386fcff04"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.514166 2832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:57:56.516387 2846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.517351 2832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:57:56.517462 2832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f25b7c34fac14ab3bfe3c58386fcff04"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.517541 2832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:56.534197 2832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:56.534866 2832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:56.535084 2832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:56.535630 2832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:56.536542 2832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:56.536592 2832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.536655 2832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:56.536680 2832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.545328 2832 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35243
I20240501 07:57:56.545368 2959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35243 every 8 connection(s)
I20240501 07:57:56.546289 2832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:57:56.550794 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2832
I20240501 07:57:56.550925 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:57:56.553100 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:56.555622 2960 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:56.555838 2960 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:56.556308 2960 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:56.557251 2247 ts_manager.cc:194] Registered new tserver with Master: f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133:35243)
I20240501 07:57:56.557801 2247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:60947
W20240501 07:57:56.729285 2963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:56.729538 2963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:56.729617 2963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:56.733481 2963 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:56.733577 2963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:56.733709 2963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:57:56.739317 2963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.2963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:56.740550 2963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:56.741883 2963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:56.751406 2971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.752555 2968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.753768 2969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.754657 2963 server_base.cc:1034] running on GCE node
I20240501 07:57:56.755131 2963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:56.755753 2963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:56.756891 2963 hybrid_clock.cc:648] HybridClock initialized: now 1714550276756873 us; error 34 us; skew 500 ppm
I20240501 07:57:56.758967 2963 webserver.cc:460] Webserver started at http://127.0.99.134:45595/ using document root <none> and password file <none>
I20240501 07:57:56.759583 2963 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:56.759660 2963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:56.759867 2963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:56.761651 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "827b6164a742460a8c2086c5cb1222f7"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.762172 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "827b6164a742460a8c2086c5cb1222f7"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.766314 2963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 07:57:56.768734 2977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.769868 2963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:57:56.769997 2963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "827b6164a742460a8c2086c5cb1222f7"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.770089 2963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:56.787354 2963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:56.788087 2963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:56.788285 2963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:56.788890 2963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:56.789927 2963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:56.789983 2963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.790052 2963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:56.790087 2963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.800743 2963 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37621
I20240501 07:57:56.800840 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37621 every 8 connection(s)
I20240501 07:57:56.801796 2963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:57:56.810732 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 2963
I20240501 07:57:56.810863 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:57:56.811714 3091 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:56.811977 3091 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:56.812494 3091 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:56.813139 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:57:56.813639 2247 ts_manager.cc:194] Registered new tserver with Master: 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134:37621)
I20240501 07:57:56.814363 2247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34521
I20240501 07:57:56.828411 2436 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
W20240501 07:57:56.928063 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:57:56.928289 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:57:56.928359 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:57:56.931517 3094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:57:56.931602 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:57:56.931674 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:57:56.935428 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:57:56.936318 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:57:56.937405 3094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:57:56.945842 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:57:56.946664 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.947903 3094 server_base.cc:1034] running on GCE node
W20240501 07:57:56.948895 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:57:56.949308 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:57:56.949878 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:57:56.951077 3094 hybrid_clock.cc:648] HybridClock initialized: now 1714550276951060 us; error 31 us; skew 500 ppm
I20240501 07:57:56.958374 3094 webserver.cc:460] Webserver started at http://127.0.99.135:34195/ using document root <none> and password file <none>
I20240501 07:57:56.959259 3094 fs_manager.cc:359] Metadata directory not provided
I20240501 07:57:56.959381 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:57:56.959645 3094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:57:56.962006 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f750f74c5d5b4295be7b3b3447f97797"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.962625 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f750f74c5d5b4295be7b3b3447f97797"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.966917 3094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 07:57:56.969831 3108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:56.971206 3094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:57:56.971315 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f750f74c5d5b4295be7b3b3447f97797"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:57:56.971403 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:57:57.002593 3094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:57:57.003436 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:57:57.003607 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:57:57.004146 3094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:57:57.005069 3094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:57:57.005122 3094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:57.005189 3094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:57:57.005221 3094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:57:57.005915 2567 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:57:57.040019 3094 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45899
I20240501 07:57:57.040397 3221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45899 every 8 connection(s)
I20240501 07:57:57.041427 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:57:57.050318 3222 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:57:57.050552 3222 heartbeater.cc:464] Registering TS with master...
I20240501 07:57:57.050993 3222 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:57:57.051401 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3094
I20240501 07:57:57.051518 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:57:57.052099 2247 ts_manager.cc:194] Registered new tserver with Master: f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135:45899)
I20240501 07:57:57.052750 2247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59515
I20240501 07:57:57.053660 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:57:57.062039 2248 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:57:57.078377 2894 tablet_service.cc:1467] Processing CreateTablet for tablet 7879c400867a4e4cbd2e51d323f78794 (DEFAULT_TABLE table=TestTable [id=0afe7693a2a24bd2a503e276cc4610d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:57:57.079257 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7879c400867a4e4cbd2e51d323f78794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:57:57.084249 3236 tablet_bootstrap.cc:492] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap starting.
I20240501 07:57:57.086699 3236 tablet_bootstrap.cc:654] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Neither blocks nor log segments found. Creating new log.
I20240501 07:57:57.087420 3236 log.cc:824] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Log is configured to *not* fsync() on all Append() calls
I20240501 07:57:57.089108 3236 tablet_bootstrap.cc:492] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: No bootstrap required, opened a new log
I20240501 07:57:57.089291 3236 ts_tablet_manager.cc:1367] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:57:57.091743 3236 raft_consensus.cc:348] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:57:57.091913 3236 raft_consensus.cc:374] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:57:57.091946 3236 raft_consensus.cc:729] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Initialized, Role: FOLLOWER
I20240501 07:57:57.092298 3236 consensus_queue.cc:260] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:57:57.092406 3236 raft_consensus.cc:388] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:57:57.092448 3236 raft_consensus.cc:482] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:57:57.092523 3236 raft_consensus.cc:3037] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:57:57.139248 3236 raft_consensus.cc:504] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:57:57.139715 3236 leader_election.cc:302] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25b7c34fac14ab3bfe3c58386fcff04; no voters:
I20240501 07:57:57.140074 3236 leader_election.cc:288] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:57:57.140216 3238 raft_consensus.cc:2781] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:57:57.140486 3238 raft_consensus.cc:686] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 LEADER]: Becoming Leader. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Running, Role: LEADER
I20240501 07:57:57.140853 3238 consensus_queue.cc:237] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:57:57.141023 2960 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:57:57.141081 3236 ts_tablet_manager.cc:1398] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20240501 07:57:57.144488 2248 catalog_manager.cc:5484] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 reported cstate change: term changed from 0 to 1, leader changed from <none> to f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:57:57.152520 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:57:57.153908 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bf25fbb990fb46ccb496debdcc7cf91e to finish bootstrapping
I20240501 07:57:57.159911 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fa963b55f7f04fdb9bd343e770f13dd5 to finish bootstrapping
I20240501 07:57:57.165627 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fe0605ab798147669ec83b685fc7ace0 to finish bootstrapping
I20240501 07:57:57.170815 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3199fb0a9d4d448687cd049ae66601c9 to finish bootstrapping
I20240501 07:57:57.176535 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f25b7c34fac14ab3bfe3c58386fcff04 to finish bootstrapping
I20240501 07:57:57.177529 2698 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:57:57.181516 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 827b6164a742460a8c2086c5cb1222f7 to finish bootstrapping
I20240501 07:57:57.187489 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f750f74c5d5b4295be7b3b3447f97797 to finish bootstrapping
I20240501 07:57:57.193501 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2570
I20240501 07:57:57.198730 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2701
I20240501 07:57:57.204223 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2832
I20240501 07:57:57.209934 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2963
I20240501 07:57:57.215363 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3094
I20240501 07:58:00.972164 2248 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:00.984062 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4663e0650c4209bbb88e87291af937 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4119ed7f60404f07b49c27fc5727c0be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:00.984073 2369 tablet_service.cc:1467] Processing CreateTablet for tablet 914df5ce34d5464a9a3876fb0cc472f1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4119ed7f60404f07b49c27fc5727c0be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:00.984062 2370 tablet_service.cc:1467] Processing CreateTablet for tablet f920c1cf13e44a8fa8c167c5299c796f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4119ed7f60404f07b49c27fc5727c0be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:00.985026 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4663e0650c4209bbb88e87291af937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:00.985038 2369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914df5ce34d5464a9a3876fb0cc472f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:00.985404 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f920c1cf13e44a8fa8c167c5299c796f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:00.990279 3258 tablet_bootstrap.cc:492] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:00.990542 3259 tablet_bootstrap.cc:492] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:00.992650 3259 tablet_bootstrap.cc:654] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:00.992650 3258 tablet_bootstrap.cc:654] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:00.993485 3258 log.cc:824] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:00.993486 3259 log.cc:824] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:00.995319 3258 tablet_bootstrap.cc:492] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:00.995319 3259 tablet_bootstrap.cc:492] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:00.995496 3259 ts_tablet_manager.cc:1367] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:58:00.995496 3258 ts_tablet_manager.cc:1367] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:58:00.999418 3258 raft_consensus.cc:348] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:00.999418 3259 raft_consensus.cc:348] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:00.999642 3259 raft_consensus.cc:374] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:00.999642 3258 raft_consensus.cc:374] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:00.999706 3258 raft_consensus.cc:729] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:00.999706 3259 raft_consensus.cc:729] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.000172 3258 consensus_queue.cc:260] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.000172 3259 consensus_queue.cc:260] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.000324 3258 raft_consensus.cc:388] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.000324 3259 raft_consensus.cc:388] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.000392 3259 raft_consensus.cc:482] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.000391 3258 raft_consensus.cc:482] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.000540 3258 raft_consensus.cc:3037] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.000540 3259 raft_consensus.cc:3037] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.001626 3258 raft_consensus.cc:504] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.001626 3259 raft_consensus.cc:504] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.002019 3259 leader_election.cc:302] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.002019 3258 leader_election.cc:302] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.002365 3259 leader_election.cc:288] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.002365 3258 leader_election.cc:288] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.002460 3263 raft_consensus.cc:2781] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.002689 3263 raft_consensus.cc:686] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.003000 3263 consensus_queue.cc:237] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.003293 3258 ts_tablet_manager.cc:1398] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 07:58:01.003654 3259 ts_tablet_manager.cc:1398] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 07:58:01.003690 3265 raft_consensus.cc:2781] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.003865 3265 raft_consensus.cc:686] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.004005 3259 tablet_bootstrap.cc:492] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.004124 3265 consensus_queue.cc:237] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.005076 3259 tablet_bootstrap.cc:654] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.006232 3259 tablet_bootstrap.cc:492] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.006319 3259 ts_tablet_manager.cc:1367] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.006354 2248 catalog_manager.cc:5484] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
I20240501 07:58:01.006848 3259 raft_consensus.cc:348] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.006954 3259 raft_consensus.cc:374] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.007028 3259 raft_consensus.cc:729] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.007177 3259 consensus_queue.cc:260] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.007257 3259 raft_consensus.cc:388] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.007288 3259 raft_consensus.cc:482] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.007342 3259 raft_consensus.cc:3037] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.008262 3259 raft_consensus.cc:504] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.008415 3259 leader_election.cc:302] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.008519 3259 leader_election.cc:288] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.008915 2247 catalog_manager.cc:5484] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
I20240501 07:58:01.009446 3262 raft_consensus.cc:2781] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.009543 3262 raft_consensus.cc:686] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.009613 3259 ts_tablet_manager.cc:1398] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:01.009644 3262 consensus_queue.cc:237] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.011924 2247 catalog_manager.cc:5484] T f920c1cf13e44a8fa8c167c5299c796f P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
W20240501 07:58:01.119812 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:01.120043 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:01.123009 3274 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 61.626 ms
time per row: 0.030813 ms
I20240501 07:58:01.225865 2247 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:01.231240 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6eab741d4248c09234574cfe48fb2c (DEFAULT_TABLE table=rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.231487 2370 tablet_service.cc:1467] Processing CreateTablet for tablet b6c35e3ea72543f1bd874e72f708cdf2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.231457 2500 tablet_service.cc:1467] Processing CreateTablet for tablet 96af3865d3ef469ea5b24cba20519bb3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.231547 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6eab741d4248c09234574cfe48fb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.231734 2500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96af3865d3ef469ea5b24cba20519bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.231734 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c35e3ea72543f1bd874e72f708cdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.233417 3258 tablet_bootstrap.cc:492] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.233714 3259 tablet_bootstrap.cc:492] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.234345 3258 tablet_bootstrap.cc:654] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.234483 3259 tablet_bootstrap.cc:654] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.235458 3258 tablet_bootstrap.cc:492] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.235513 3259 tablet_bootstrap.cc:492] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.235534 3258 ts_tablet_manager.cc:1367] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.235579 3259 ts_tablet_manager.cc:1367] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:01.235978 3259 raft_consensus.cc:348] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.236064 3259 raft_consensus.cc:374] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.236024 3258 raft_consensus.cc:348] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.236088 3259 raft_consensus.cc:729] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.236111 3258 raft_consensus.cc:374] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.236155 3258 raft_consensus.cc:729] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.236291 3258 consensus_queue.cc:260] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.236276 3259 consensus_queue.cc:260] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.236371 3258 raft_consensus.cc:388] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.236383 3259 raft_consensus.cc:388] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.236413 3258 raft_consensus.cc:482] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.236434 3259 raft_consensus.cc:482] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.236469 3258 raft_consensus.cc:3037] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.236480 3259 raft_consensus.cc:3037] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.237084 3258 raft_consensus.cc:504] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.237203 3258 leader_election.cc:302] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.237303 3258 leader_election.cc:288] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.237350 3259 raft_consensus.cc:504] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.237432 3266 raft_consensus.cc:2781] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.237488 3259 leader_election.cc:302] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.237507 3266 raft_consensus.cc:686] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.237540 3258 ts_tablet_manager.cc:1398] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.237614 3266 consensus_queue.cc:237] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.237847 3259 leader_election.cc:288] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.237896 3262 raft_consensus.cc:2781] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.237967 3262 raft_consensus.cc:686] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.237673 3258 tablet_bootstrap.cc:492] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.238076 3262 consensus_queue.cc:237] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.238890 3258 tablet_bootstrap.cc:654] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.239406 2247 catalog_manager.cc:5484] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
I20240501 07:58:01.240070 3259 ts_tablet_manager.cc:1398] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:01.240425 2248 catalog_manager.cc:5484] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
I20240501 07:58:01.240675 3258 tablet_bootstrap.cc:492] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.241149 3258 ts_tablet_manager.cc:1367] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:01.241588 3258 raft_consensus.cc:348] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.241677 3258 raft_consensus.cc:374] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.241717 3258 raft_consensus.cc:729] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.241845 3258 consensus_queue.cc:260] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.241942 3258 raft_consensus.cc:388] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.241998 3258 raft_consensus.cc:482] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.242036 3258 raft_consensus.cc:3037] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.242802 3258 raft_consensus.cc:504] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.242925 3258 leader_election.cc:302] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.243098 3266 raft_consensus.cc:2781] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.243178 3266 raft_consensus.cc:686] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.243289 3266 consensus_queue.cc:237] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.243635 3258 leader_election.cc:288] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.243865 3258 ts_tablet_manager.cc:1398] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:01.244361 2247 catalog_manager.cc:5484] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
W20240501 07:58:01.355054 3316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:01.355304 3316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:01.358307 3316 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 71.3405 ms
time per row: 0.0356703 ms
I20240501 07:58:01.470003 2247 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:01.475629 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb7d8d7017f48be83670a06fc02b26d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c723405d23a94457ac618ebd7a319e96]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.475761 2500 tablet_service.cc:1467] Processing CreateTablet for tablet d6be2584c472449c8d570057f24d4ee4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c723405d23a94457ac618ebd7a319e96]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.475880 2369 tablet_service.cc:1467] Processing CreateTablet for tablet 30d99a0709f345848780a56d85a65186 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c723405d23a94457ac618ebd7a319e96]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.475962 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb7d8d7017f48be83670a06fc02b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.476059 2500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6be2584c472449c8d570057f24d4ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.476135 2369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d99a0709f345848780a56d85a65186. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.477896 3259 tablet_bootstrap.cc:492] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.478212 3258 tablet_bootstrap.cc:492] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.478688 3259 tablet_bootstrap.cc:654] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.478984 3258 tablet_bootstrap.cc:654] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.479899 3259 tablet_bootstrap.cc:492] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.479986 3259 ts_tablet_manager.cc:1367] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:01.480293 3258 tablet_bootstrap.cc:492] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.480367 3258 ts_tablet_manager.cc:1367] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:01.480360 3259 raft_consensus.cc:348] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.480469 3259 raft_consensus.cc:374] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.480512 3259 raft_consensus.cc:729] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.480648 3259 consensus_queue.cc:260] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.480726 3259 raft_consensus.cc:388] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.480770 3259 raft_consensus.cc:482] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.480819 3259 raft_consensus.cc:3037] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.480830 3258 raft_consensus.cc:348] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.480945 3258 raft_consensus.cc:374] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.480986 3258 raft_consensus.cc:729] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.481120 3258 consensus_queue.cc:260] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.481200 3258 raft_consensus.cc:388] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.481226 3258 raft_consensus.cc:482] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.481274 3258 raft_consensus.cc:3037] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.481577 3259 raft_consensus.cc:504] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.481700 3259 leader_election.cc:302] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.481796 3259 leader_election.cc:288] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.481878 3265 raft_consensus.cc:2781] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.481971 3258 raft_consensus.cc:504] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.482023 3265 raft_consensus.cc:686] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.482043 3259 ts_tablet_manager.cc:1398] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:01.482180 3259 tablet_bootstrap.cc:492] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.482143 3265 consensus_queue.cc:237] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.482327 3258 leader_election.cc:302] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.482470 3258 leader_election.cc:288] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.482517 3266 raft_consensus.cc:2781] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.483115 3258 ts_tablet_manager.cc:1398] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:01.483299 3266 raft_consensus.cc:686] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.483436 3266 consensus_queue.cc:237] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.483510 2245 catalog_manager.cc:5484] T 30d99a0709f345848780a56d85a65186 P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
I20240501 07:58:01.484056 3259 tablet_bootstrap.cc:654] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.485025 2247 catalog_manager.cc:5484] T d6be2584c472449c8d570057f24d4ee4 P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
I20240501 07:58:01.486136 3259 tablet_bootstrap.cc:492] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.486266 3259 ts_tablet_manager.cc:1367] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:01.486718 3259 raft_consensus.cc:348] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.486840 3259 raft_consensus.cc:374] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.486891 3259 raft_consensus.cc:729] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.487030 3259 consensus_queue.cc:260] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.487118 3259 raft_consensus.cc:388] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.487160 3259 raft_consensus.cc:482] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.487221 3259 raft_consensus.cc:3037] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.487910 3259 raft_consensus.cc:504] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.488046 3259 leader_election.cc:302] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.488147 3259 leader_election.cc:288] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.488201 3265 raft_consensus.cc:2781] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.488289 3265 raft_consensus.cc:686] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.488359 3259 ts_tablet_manager.cc:1398] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:01.488389 3265 consensus_queue.cc:237] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.489766 2247 catalog_manager.cc:5484] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
W20240501 07:58:01.603107 3350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:01.603375 3350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:01.606407 3350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 62.6973 ms
time per row: 0.0313486 ms
I20240501 07:58:01.710669 2247 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:01.716113 2500 tablet_service.cc:1467] Processing CreateTablet for tablet e734066a97d145338ab9c525d5d4c1dc (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.716171 2501 tablet_service.cc:1467] Processing CreateTablet for tablet c7c7acd9cd91461197fc2e2c86bd570f (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.716382 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 56c2c5346ceb43a0b74ef2689e2d3d6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.716463 2500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e734066a97d145338ab9c525d5d4c1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.716645 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c2c5346ceb43a0b74ef2689e2d3d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.718660 3259 tablet_bootstrap.cc:492] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.719164 3258 tablet_bootstrap.cc:492] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.719496 3259 tablet_bootstrap.cc:654] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.720067 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c7acd9cd91461197fc2e2c86bd570f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.720077 3258 tablet_bootstrap.cc:654] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.721117 3258 tablet_bootstrap.cc:492] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.721184 3258 ts_tablet_manager.cc:1367] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:01.721535 3258 raft_consensus.cc:348] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.721624 3258 raft_consensus.cc:374] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.721658 3258 raft_consensus.cc:729] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.721776 3258 consensus_queue.cc:260] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.721841 3258 raft_consensus.cc:388] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.721899 3258 raft_consensus.cc:482] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.721951 3258 raft_consensus.cc:3037] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.721956 3259 tablet_bootstrap.cc:492] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.722038 3259 ts_tablet_manager.cc:1367] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:01.722417 3259 raft_consensus.cc:348] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.722513 3259 raft_consensus.cc:374] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.722556 3259 raft_consensus.cc:729] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.722698 3259 consensus_queue.cc:260] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.722712 3258 raft_consensus.cc:504] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.722770 3259 raft_consensus.cc:388] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.722793 3259 raft_consensus.cc:482] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.722821 3259 raft_consensus.cc:3037] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.722833 3258 leader_election.cc:302] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.722924 3258 leader_election.cc:288] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.722999 3263 raft_consensus.cc:2781] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.723106 3263 raft_consensus.cc:686] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.723208 3263 consensus_queue.cc:237] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.723438 3258 ts_tablet_manager.cc:1398] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.723606 3258 tablet_bootstrap.cc:492] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.723721 3259 raft_consensus.cc:504] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.723826 3259 leader_election.cc:302] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.723904 3259 leader_election.cc:288] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.723979 3262 raft_consensus.cc:2781] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.724052 3262 raft_consensus.cc:686] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.724135 3259 ts_tablet_manager.cc:1398] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.724153 3262 consensus_queue.cc:237] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.725230 3258 tablet_bootstrap.cc:654] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.725522 2245 catalog_manager.cc:5484] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
I20240501 07:58:01.725522 2247 catalog_manager.cc:5484] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
I20240501 07:58:01.726410 3258 tablet_bootstrap.cc:492] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.726492 3258 ts_tablet_manager.cc:1367] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:01.727090 3258 raft_consensus.cc:348] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.727176 3258 raft_consensus.cc:374] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.727213 3258 raft_consensus.cc:729] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.727339 3258 consensus_queue.cc:260] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.727402 3258 raft_consensus.cc:388] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.727448 3258 raft_consensus.cc:482] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.727492 3258 raft_consensus.cc:3037] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.728246 3258 raft_consensus.cc:504] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.728397 3258 leader_election.cc:302] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.728492 3258 leader_election.cc:288] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.728549 3264 raft_consensus.cc:2781] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.728613 3264 raft_consensus.cc:686] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.728722 3258 ts_tablet_manager.cc:1398] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:01.728711 3264 consensus_queue.cc:237] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.730552 2245 catalog_manager.cc:5484] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
W20240501 07:58:01.845697 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:01.845965 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:01.848961 3383 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 65.6582 ms
time per row: 0.0328291 ms
I20240501 07:58:01.957017 2244 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58522:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:01.964519 2370 tablet_service.cc:1467] Processing CreateTablet for tablet eff829f74ea44c7c9cefa72260ac2165 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d541bdfe4e0042b8b8361aa1e34290d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.964561 2369 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0be484474f4cde809c8757a87c1bea (DEFAULT_TABLE table=rebalance_test_table_4 [id=d541bdfe4e0042b8b8361aa1e34290d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.964548 2501 tablet_service.cc:1467] Processing CreateTablet for tablet fe5420569ff445debb0e2ffdc7c4b1d1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d541bdfe4e0042b8b8361aa1e34290d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:01.964859 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff829f74ea44c7c9cefa72260ac2165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.964861 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5420569ff445debb0e2ffdc7c4b1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.965261 2369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0be484474f4cde809c8757a87c1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:01.966642 3259 tablet_bootstrap.cc:492] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.967042 3258 tablet_bootstrap.cc:492] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5: Bootstrap starting.
I20240501 07:58:01.967620 3259 tablet_bootstrap.cc:654] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.967913 3258 tablet_bootstrap.cc:654] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.968744 3259 tablet_bootstrap.cc:492] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.968833 3259 ts_tablet_manager.cc:1367] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.969229 3259 raft_consensus.cc:348] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.969322 3259 raft_consensus.cc:374] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.969357 3259 raft_consensus.cc:729] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.969496 3259 consensus_queue.cc:260] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.969578 3259 raft_consensus.cc:388] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.969621 3259 raft_consensus.cc:482] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.969672 3259 raft_consensus.cc:3037] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.970427 3259 raft_consensus.cc:504] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.970554 3259 leader_election.cc:302] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.970654 3259 leader_election.cc:288] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.970738 3267 raft_consensus.cc:2781] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.970834 3267 raft_consensus.cc:686] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.970893 3259 ts_tablet_manager.cc:1398] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:01.970943 3267 consensus_queue.cc:237] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.971091 3259 tablet_bootstrap.cc:492] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Bootstrap starting.
I20240501 07:58:01.971735 3258 tablet_bootstrap.cc:492] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5: No bootstrap required, opened a new log
I20240501 07:58:01.971848 3258 ts_tablet_manager.cc:1367] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:01.972316 3258 raft_consensus.cc:348] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.972357 2244 catalog_manager.cc:5484] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
I20240501 07:58:01.972438 3258 raft_consensus.cc:374] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.972451 3259 tablet_bootstrap.cc:654] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:01.972476 3258 raft_consensus.cc:729] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.972609 3258 consensus_queue.cc:260] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.972702 3258 raft_consensus.cc:388] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.972743 3258 raft_consensus.cc:482] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.972801 3258 raft_consensus.cc:3037] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.973713 3258 raft_consensus.cc:504] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.973853 3258 leader_election.cc:302] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:01.974005 3258 leader_election.cc:288] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.974045 3266 raft_consensus.cc:2781] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.974112 3266 raft_consensus.cc:686] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Leader. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:01.974218 3266 consensus_queue.cc:237] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:01.974362 3258 ts_tablet_manager.cc:1398] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:01.975075 3259 tablet_bootstrap.cc:492] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: No bootstrap required, opened a new log
I20240501 07:58:01.975165 3259 ts_tablet_manager.cc:1367] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:01.975625 3259 raft_consensus.cc:348] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.975589 2244 catalog_manager.cc:5484] T fe5420569ff445debb0e2ffdc7c4b1d1 P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } }
I20240501 07:58:01.975706 3259 raft_consensus.cc:374] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:01.975740 3259 raft_consensus.cc:729] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Initialized, Role: FOLLOWER
I20240501 07:58:01.975854 3259 consensus_queue.cc:260] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.975924 3259 raft_consensus.cc:388] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:01.975965 3259 raft_consensus.cc:482] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:01.976009 3259 raft_consensus.cc:3037] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:01.976783 3259 raft_consensus.cc:504] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.976902 3259 leader_election.cc:302] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:01.976994 3259 leader_election.cc:288] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:01.977246 3259 ts_tablet_manager.cc:1398] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:01.977432 3262 raft_consensus.cc:2781] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:01.977509 3262 raft_consensus.cc:686] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Leader. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:01.977618 3262 consensus_queue.cc:237] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:01.979091 2244 catalog_manager.cc:5484] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } }
W20240501 07:58:02.091302 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:02.091567 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:02.094599 3415 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 59.9366 ms
time per row: 0.0299683 ms
I20240501 07:58:02.201668 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:39669
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:33335
--webserver_port=34307
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:58:02.310817 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:02.311084 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:02.311139 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:02.314167 3441 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:02.314230 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:02.314304 3441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:02.317890 3441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:39669
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=34307
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:02.318822 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:02.320065 3441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:02.329290 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.330415 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.331444 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:02.332370 3441 server_base.cc:1034] running on GCE node
I20240501 07:58:02.332751 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:02.333333 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:02.334477 3441 hybrid_clock.cc:648] HybridClock initialized: now 1714550282334462 us; error 29 us; skew 500 ppm
I20240501 07:58:02.336421 3441 webserver.cc:460] Webserver started at http://127.0.99.131:34307/ using document root <none> and password file <none>
I20240501 07:58:02.336972 3441 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:02.337044 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:02.340308 3441 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 07:58:02.342063 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.343127 3441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 07:58:02.343230 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fe0605ab798147669ec83b685fc7ace0"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:58:02.343657 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:02.362433 3441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:02.363142 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:02.363299 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:02.363832 3441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:02.364702 3441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:02.364750 3441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.364813 3441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:02.364840 3441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.373692 3441 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39669
I20240501 07:58:02.373735 3568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39669 every 8 connection(s)
I20240501 07:58:02.374724 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:02.377357 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3441
I20240501 07:58:02.377897 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:35435
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:33335
--webserver_port=39639
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:02.384320 3569 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:02.384562 3569 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:02.385030 3569 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:02.386118 2244 ts_manager.cc:194] Re-registered known tserver with Master: fe0605ab798147669ec83b685fc7ace0 (127.0.99.131:39669)
I20240501 07:58:02.386682 2244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:42783
W20240501 07:58:02.492996 3572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:02.493214 3572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:02.493284 3572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:02.496387 3572 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:02.496465 3572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:02.496556 3572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:02.500283 3572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:35435
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=39639
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:02.501204 3572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:02.502362 3572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:02.510978 3577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.511821 3578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.512528 3580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:02.513803 3572 server_base.cc:1034] running on GCE node
I20240501 07:58:02.514230 3572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:02.514793 3572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:02.515939 3572 hybrid_clock.cc:648] HybridClock initialized: now 1714550282515909 us; error 50 us; skew 500 ppm
I20240501 07:58:02.517848 3572 webserver.cc:460] Webserver started at http://127.0.99.132:39639/ using document root <none> and password file <none>
I20240501 07:58:02.518411 3572 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:02.518477 3572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:02.521734 3572 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:02.523563 3586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.524585 3572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:02.524683 3572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3199fb0a9d4d448687cd049ae66601c9"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:58:02.525084 3572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:02.558161 3572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:02.558799 3572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:02.558936 3572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:02.559453 3572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:02.560461 3572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:02.560510 3572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.560550 3572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:02.560565 3572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.569530 3572 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:35435
I20240501 07:58:02.569564 3699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:35435 every 8 connection(s)
I20240501 07:58:02.570569 3572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:02.574791 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3572
I20240501 07:58:02.575273 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:35243
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:33335
--webserver_port=39913
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:02.579914 3700 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:02.580152 3700 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:02.580619 3700 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:02.581548 2244 ts_manager.cc:194] Re-registered known tserver with Master: 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132:35435)
I20240501 07:58:02.582247 2244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:43795
W20240501 07:58:02.684487 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:02.684681 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:02.684723 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:02.687800 3703 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:02.687857 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:02.687927 3703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:02.691558 3703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:35243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=39913
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:02.692423 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:02.693456 3703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:02.702025 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.702944 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.704098 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:02.705160 3703 server_base.cc:1034] running on GCE node
I20240501 07:58:02.705534 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:02.706102 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:02.707275 3703 hybrid_clock.cc:648] HybridClock initialized: now 1714550282707262 us; error 26 us; skew 500 ppm
I20240501 07:58:02.709290 3703 webserver.cc:460] Webserver started at http://127.0.99.133:39913/ using document root <none> and password file <none>
I20240501 07:58:02.709827 3703 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:02.709909 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:02.713158 3703 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:02.715096 3717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.716054 3703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:02.716166 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f25b7c34fac14ab3bfe3c58386fcff04"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:58:02.716562 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:02.737166 3703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:02.737800 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:02.737958 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:02.738492 3703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:02.739665 3724 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:58:02.742738 3703 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:58:02.742806 3703 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 07:58:02.742895 3703 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:58:02.744414 3703 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:58:02.744457 3703 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:02.744748 3724 tablet_bootstrap.cc:492] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap starting.
I20240501 07:58:02.754420 3703 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35243
I20240501 07:58:02.754446 3831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35243 every 8 connection(s)
I20240501 07:58:02.755472 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:02.758955 3724 log.cc:824] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:02.761076 3724 tablet_bootstrap.cc:492] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:02.761446 3724 tablet_bootstrap.cc:492] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap complete.
I20240501 07:58:02.762635 3724 ts_tablet_manager.cc:1367] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20240501 07:58:02.762828 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3703
I20240501 07:58:02.763291 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:37621
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:33335
--webserver_port=45595
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:02.763648 3832 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:02.763876 3832 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:02.764310 3832 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:02.765805 3724 raft_consensus.cc:348] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:02.766178 2244 ts_manager.cc:194] Re-registered known tserver with Master: f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133:35243)
I20240501 07:58:02.766191 3724 raft_consensus.cc:729] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Initialized, Role: FOLLOWER
I20240501 07:58:02.766667 3724 consensus_queue.cc:260] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:02.766798 3724 raft_consensus.cc:388] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:02.766857 3724 raft_consensus.cc:482] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:02.766952 3724 raft_consensus.cc:3037] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:02.767191 2244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50195
I20240501 07:58:02.768553 3724 raft_consensus.cc:504] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:02.768848 3724 leader_election.cc:302] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25b7c34fac14ab3bfe3c58386fcff04; no voters:
I20240501 07:58:02.769098 3724 leader_election.cc:288] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 07:58:02.769243 3837 raft_consensus.cc:2781] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:02.769445 3837 raft_consensus.cc:686] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 LEADER]: Becoming Leader. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Running, Role: LEADER
I20240501 07:58:02.769740 3837 consensus_queue.cc:237] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:02.769802 3832 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:02.769910 3724 ts_tablet_manager.cc:1398] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
I20240501 07:58:02.771759 2244 catalog_manager.cc:5484] T 7879c400867a4e4cbd2e51d323f78794 P f25b7c34fac14ab3bfe3c58386fcff04 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
W20240501 07:58:02.871510 3836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:02.871747 3836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:02.871816 3836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:02.874893 3836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:02.874975 3836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:02.875098 3836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:02.878944 3836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:37621
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=45595
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:02.879904 3836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:02.881049 3836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:02.890098 3849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.891181 3850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:02.892046 3852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:02.892876 3836 server_base.cc:1034] running on GCE node
I20240501 07:58:02.893296 3836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:02.893875 3836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:02.895074 3836 hybrid_clock.cc:648] HybridClock initialized: now 1714550282895053 us; error 33 us; skew 500 ppm
I20240501 07:58:02.897070 3836 webserver.cc:460] Webserver started at http://127.0.99.134:45595/ using document root <none> and password file <none>
I20240501 07:58:02.897639 3836 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:02.897716 3836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:02.901167 3836 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:02.902997 3858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.904036 3836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:02.904138 3836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "827b6164a742460a8c2086c5cb1222f7"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:58:02.904533 3836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:02.920979 3836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:02.921624 3836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:02.921805 3836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:02.922345 3836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:02.923316 3836 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:02.923367 3836 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.923434 3836 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:02.923462 3836 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:02.932158 3836 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37621
I20240501 07:58:02.932216 3971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37621 every 8 connection(s)
I20240501 07:58:02.933183 3836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:02.939596 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3836
I20240501 07:58:02.939977 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:45899
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:33335
--webserver_port=34195
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:02.941769 3972 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:02.942019 3972 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:02.942498 3972 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:02.943547 2244 ts_manager.cc:194] Re-registered known tserver with Master: 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134:37621)
I20240501 07:58:02.944118 2244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:54911
W20240501 07:58:03.053609 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:03.053817 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:03.053884 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:03.056993 3975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:03.057070 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:03.057157 3975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:03.060657 3975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:45899
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=34195
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.3975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:03.061498 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:03.062567 3975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:03.071135 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:03.071792 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:03.072994 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:03.074090 3975 server_base.cc:1034] running on GCE node
I20240501 07:58:03.074504 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:03.075101 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:03.076272 3975 hybrid_clock.cc:648] HybridClock initialized: now 1714550283076259 us; error 26 us; skew 500 ppm
I20240501 07:58:03.078136 3975 webserver.cc:460] Webserver started at http://127.0.99.135:34195/ using document root <none> and password file <none>
I20240501 07:58:03.078640 3975 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:03.078706 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:03.081912 3975 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:03.083796 3989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:03.084820 3975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:03.084921 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f750f74c5d5b4295be7b3b3447f97797"
format_stamp: "Formatted at 2024-05-01 07:57:56 on dist-test-slave-qm51"
I20240501 07:58:03.085328 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:03.116226 3975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:03.116911 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:03.117094 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:03.117656 3975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:03.118562 3975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:03.118618 3975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:03.118678 3975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:03.118710 3975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:03.127512 3975 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45899
I20240501 07:58:03.127545 4102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45899 every 8 connection(s)
I20240501 07:58:03.128543 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:03.136791 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 3975
I20240501 07:58:03.137442 4103 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:03.137663 4103 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:03.138176 4103 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:03.139199 2244 ts_manager.cc:194] Re-registered known tserver with Master: f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135:45899)
I20240501 07:58:03.139709 2244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:53457
I20240501 07:58:03.331564 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.332121 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:03.338210 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.342499 2501 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:03.345942 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:03.352226 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.353220 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.388674 3569 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:03.477366 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.480301 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.481369 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.485803 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:03.486042 2501 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:03.496485 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:03.511657 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:03.550153 2389 consensus_queue.cc:237] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
W20240501 07:58:03.555318 2326 consensus_peers.cc:485] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e -> Peer 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134:37621): Couldn't send request to peer 827b6164a742460a8c2086c5cb1222f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 914df5ce34d5464a9a3876fb0cc472f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:03.584270 3700 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:03.946501 3972 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:04.018569 4205 ts_tablet_manager.cc:902] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:04.019397 4205 tablet_copy_client.cc:308] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:04.023056 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet 914df5ce34d5464a9a3876fb0cc472f1 from peer 827b6164a742460a8c2086c5cb1222f7 ({username='slave'} at 127.0.99.134:35729)
I20240501 07:58:04.023190 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet 914df5ce34d5464a9a3876fb0cc472f1 from peer 827b6164a742460a8c2086c5cb1222f7 at {username='slave'} at 127.0.99.134:35729: session id = 827b6164a742460a8c2086c5cb1222f7-914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:04.024338 2410 tablet_copy_source_session.cc:205] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:04.027470 4205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 914df5ce34d5464a9a3876fb0cc472f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:04.032567 4205 tablet_copy_client.cc:783] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:04.032753 4205 tablet_copy_client.cc:647] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:04.034564 4205 tablet_copy_client.cc:523] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:04.036087 4205 tablet_bootstrap.cc:492] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap starting.
I20240501 07:58:04.041141 4205 log.cc:824] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:04.061115 4205 tablet_bootstrap.cc:492] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:04.061506 4205 tablet_bootstrap.cc:492] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap complete.
I20240501 07:58:04.061825 4205 ts_tablet_manager.cc:1367] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 07:58:04.063823 4205 raft_consensus.cc:348] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:04.064368 4205 raft_consensus.cc:729] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827b6164a742460a8c2086c5cb1222f7, State: Initialized, Role: FOLLOWER
I20240501 07:58:04.064644 4205 consensus_queue.cc:260] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:04.065361 4205 ts_tablet_manager.cc:1398] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 07:58:04.066105 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session 827b6164a742460a8c2086c5cb1222f7-914df5ce34d5464a9a3876fb0cc472f1 received from {username='slave'} at 127.0.99.134:35729
I20240501 07:58:04.066200 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session 827b6164a742460a8c2086c5cb1222f7-914df5ce34d5464a9a3876fb0cc472f1 on tablet 914df5ce34d5464a9a3876fb0cc472f1 with peer 827b6164a742460a8c2086c5cb1222f7
I20240501 07:58:04.141532 4103 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:04.518190 3926 raft_consensus.cc:1203] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:04.519610 4213 raft_consensus.cc:2932] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:04.520040 3926 raft_consensus.cc:2932] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:04.523586 2240 catalog_manager.cc:5484] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:04.540462 2521 consensus_queue.cc:237] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
W20240501 07:58:04.546016 2456 consensus_peers.cc:485] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 -> Peer fe0605ab798147669ec83b685fc7ace0 (127.0.99.131:39669): Couldn't send request to peer fe0605ab798147669ec83b685fc7ace0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96af3865d3ef469ea5b24cba20519bb3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:04.929071 4240 ts_tablet_manager.cc:902] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Initiating tablet copy from peer fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:04.929847 4240 tablet_copy_client.cc:308] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38583
I20240501 07:58:04.933928 2541 tablet_copy_service.cc:140] P fa963b55f7f04fdb9bd343e770f13dd5: Received BeginTabletCopySession request for tablet 96af3865d3ef469ea5b24cba20519bb3 from peer fe0605ab798147669ec83b685fc7ace0 ({username='slave'} at 127.0.99.131:41589)
I20240501 07:58:04.934089 2541 tablet_copy_service.cc:161] P fa963b55f7f04fdb9bd343e770f13dd5: Beginning new tablet copy session on tablet 96af3865d3ef469ea5b24cba20519bb3 from peer fe0605ab798147669ec83b685fc7ace0 at {username='slave'} at 127.0.99.131:41589: session id = fe0605ab798147669ec83b685fc7ace0-96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:04.935590 2541 tablet_copy_source_session.cc:205] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:04.938899 4240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96af3865d3ef469ea5b24cba20519bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:04.944283 4240 tablet_copy_client.cc:783] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:04.944471 4240 tablet_copy_client.cc:647] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:04.946208 4240 tablet_copy_client.cc:523] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:04.947849 4240 tablet_bootstrap.cc:492] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Bootstrap starting.
I20240501 07:58:04.953073 4240 log.cc:824] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:04.972477 4240 tablet_bootstrap.cc:492] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:04.972970 4240 tablet_bootstrap.cc:492] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Bootstrap complete.
I20240501 07:58:04.973294 4240 ts_tablet_manager.cc:1367] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 07:58:04.975359 4240 raft_consensus.cc:348] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:04.975945 4240 raft_consensus.cc:729] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0605ab798147669ec83b685fc7ace0, State: Initialized, Role: FOLLOWER
I20240501 07:58:04.976256 4240 consensus_queue.cc:260] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:04.977120 4240 ts_tablet_manager.cc:1398] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 07:58:04.977932 2541 tablet_copy_service.cc:342] P fa963b55f7f04fdb9bd343e770f13dd5: Request end of tablet copy session fe0605ab798147669ec83b685fc7ace0-96af3865d3ef469ea5b24cba20519bb3 received from {username='slave'} at 127.0.99.131:41589
I20240501 07:58:04.978044 2541 tablet_copy_service.cc:433] P fa963b55f7f04fdb9bd343e770f13dd5: ending tablet copy session fe0605ab798147669ec83b685fc7ace0-96af3865d3ef469ea5b24cba20519bb3 on tablet 96af3865d3ef469ea5b24cba20519bb3 with peer fe0605ab798147669ec83b685fc7ace0
I20240501 07:58:05.493060 3523 raft_consensus.cc:1203] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:05.494436 4251 raft_consensus.cc:2932] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:05.495102 3523 raft_consensus.cc:2932] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:05.497447 2247 catalog_manager.cc:5484] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:05.515569 2389 consensus_queue.cc:237] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
W20240501 07:58:05.520323 2324 consensus_peers.cc:485] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e -> Peer f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135:45899): Couldn't send request to peer f750f74c5d5b4295be7b3b3447f97797. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eff829f74ea44c7c9cefa72260ac2165. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:06.117384 4276 ts_tablet_manager.cc:902] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:06.118067 4276 tablet_copy_client.cc:308] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:06.121909 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet eff829f74ea44c7c9cefa72260ac2165 from peer f750f74c5d5b4295be7b3b3447f97797 ({username='slave'} at 127.0.99.135:33193)
I20240501 07:58:06.122022 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet eff829f74ea44c7c9cefa72260ac2165 from peer f750f74c5d5b4295be7b3b3447f97797 at {username='slave'} at 127.0.99.135:33193: session id = f750f74c5d5b4295be7b3b3447f97797-eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:06.122808 2410 tablet_copy_source_session.cc:205] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:06.125846 4276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff829f74ea44c7c9cefa72260ac2165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:06.131276 4276 tablet_copy_client.cc:783] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:06.131520 4276 tablet_copy_client.cc:647] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:06.133071 4276 tablet_copy_client.cc:523] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:06.134539 4276 tablet_bootstrap.cc:492] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Bootstrap starting.
I20240501 07:58:06.139660 4276 log.cc:824] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:06.159698 4276 tablet_bootstrap.cc:492] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:06.160084 4276 tablet_bootstrap.cc:492] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Bootstrap complete.
I20240501 07:58:06.160368 4276 ts_tablet_manager.cc:1367] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240501 07:58:06.162304 4276 raft_consensus.cc:348] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:06.162891 4276 raft_consensus.cc:729] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f750f74c5d5b4295be7b3b3447f97797, State: Initialized, Role: FOLLOWER
I20240501 07:58:06.163193 4276 consensus_queue.cc:260] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:06.164034 4276 ts_tablet_manager.cc:1398] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:06.164789 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session f750f74c5d5b4295be7b3b3447f97797-eff829f74ea44c7c9cefa72260ac2165 received from {username='slave'} at 127.0.99.135:33193
I20240501 07:58:06.164877 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session f750f74c5d5b4295be7b3b3447f97797-eff829f74ea44c7c9cefa72260ac2165 on tablet eff829f74ea44c7c9cefa72260ac2165 with peer f750f74c5d5b4295be7b3b3447f97797
I20240501 07:58:06.734390 4057 raft_consensus.cc:1203] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:06.735863 4248 raft_consensus.cc:2932] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:06.736302 4057 raft_consensus.cc:2932] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:06.737490 4248 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32852 (request call id 0) took 1222 ms. Trace:
I20240501 07:58:06.737586 4248 rpcz_store.cc:276] 0501 07:58:05.515223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:05.515278 (+ 55us) service_pool.cc:225] Handling call
0501 07:58:05.516013 (+ 735us) log.cc:842] Serialized 257 byte log entry
0501 07:58:06.737469 (+1221456us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":47,"raft.queue_time_us":55,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":155,"thread_start_us":54,"threads_started":1,"wal-append.queue_time_us":156,"wal-append.run_cpu_time_us":320,"wal-append.run_wall_time_us":1000384}
I20240501 07:58:06.738986 2240 catalog_manager.cc:5484] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:06.760295 2521 consensus_queue.cc:237] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
W20240501 07:58:06.764768 2457 consensus_peers.cc:485] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 -> Peer 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132:35435): Couldn't send request to peer 3199fb0a9d4d448687cd049ae66601c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7c7acd9cd91461197fc2e2c86bd570f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:07.333675 4313 ts_tablet_manager.cc:902] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Initiating tablet copy from peer fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:07.334410 4313 tablet_copy_client.cc:308] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38583
I20240501 07:58:07.338312 2541 tablet_copy_service.cc:140] P fa963b55f7f04fdb9bd343e770f13dd5: Received BeginTabletCopySession request for tablet c7c7acd9cd91461197fc2e2c86bd570f from peer 3199fb0a9d4d448687cd049ae66601c9 ({username='slave'} at 127.0.99.132:35185)
I20240501 07:58:07.338428 2541 tablet_copy_service.cc:161] P fa963b55f7f04fdb9bd343e770f13dd5: Beginning new tablet copy session on tablet c7c7acd9cd91461197fc2e2c86bd570f from peer 3199fb0a9d4d448687cd049ae66601c9 at {username='slave'} at 127.0.99.132:35185: session id = 3199fb0a9d4d448687cd049ae66601c9-c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:07.339334 2541 tablet_copy_source_session.cc:205] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:07.342433 4313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c7acd9cd91461197fc2e2c86bd570f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:07.347914 4313 tablet_copy_client.cc:783] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:07.348145 4313 tablet_copy_client.cc:647] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:07.349671 4313 tablet_copy_client.cc:523] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:07.351281 4313 tablet_bootstrap.cc:492] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap starting.
I20240501 07:58:07.357112 4313 log.cc:824] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:07.378382 4313 tablet_bootstrap.cc:492] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:07.378831 4313 tablet_bootstrap.cc:492] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap complete.
I20240501 07:58:07.379184 4313 ts_tablet_manager.cc:1367] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.004s
I20240501 07:58:07.381220 4313 raft_consensus.cc:348] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:07.381783 4313 raft_consensus.cc:729] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3199fb0a9d4d448687cd049ae66601c9, State: Initialized, Role: FOLLOWER
I20240501 07:58:07.382078 4313 consensus_queue.cc:260] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:07.382963 4313 ts_tablet_manager.cc:1398] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:07.383829 2541 tablet_copy_service.cc:342] P fa963b55f7f04fdb9bd343e770f13dd5: Request end of tablet copy session 3199fb0a9d4d448687cd049ae66601c9-c7c7acd9cd91461197fc2e2c86bd570f received from {username='slave'} at 127.0.99.132:35185
I20240501 07:58:07.383919 2541 tablet_copy_service.cc:433] P fa963b55f7f04fdb9bd343e770f13dd5: ending tablet copy session 3199fb0a9d4d448687cd049ae66601c9-c7c7acd9cd91461197fc2e2c86bd570f on tablet c7c7acd9cd91461197fc2e2c86bd570f with peer 3199fb0a9d4d448687cd049ae66601c9
I20240501 07:58:07.885505 3654 raft_consensus.cc:1203] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:07.886750 4324 raft_consensus.cc:2932] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:07.887303 3654 raft_consensus.cc:2932] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:07.888188 4324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40202 (request call id 0) took 1128 ms. Trace:
I20240501 07:58:07.888461 4324 rpcz_store.cc:276] 0501 07:58:06.759920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:06.759962 (+ 42us) service_pool.cc:225] Handling call
0501 07:58:06.760760 (+ 798us) log.cc:842] Serialized 257 byte log entry
0501 07:58:07.888169 (+1127409us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":40,"raft.queue_time_us":44,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":117,"wal-append.run_cpu_time_us":268,"wal-append.run_wall_time_us":1000333}
I20240501 07:58:07.890697 2247 catalog_manager.cc:5484] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:07.912087 2389 consensus_queue.cc:237] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
W20240501 07:58:07.916751 2325 consensus_peers.cc:485] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e -> Peer f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133:35243): Couldn't send request to peer f25b7c34fac14ab3bfe3c58386fcff04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eb7d8d7017f48be83670a06fc02b26d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:08.534875 4351 ts_tablet_manager.cc:902] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:08.535569 4351 tablet_copy_client.cc:308] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:08.539072 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet 6eb7d8d7017f48be83670a06fc02b26d from peer f25b7c34fac14ab3bfe3c58386fcff04 ({username='slave'} at 127.0.99.133:34243)
I20240501 07:58:08.539173 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet 6eb7d8d7017f48be83670a06fc02b26d from peer f25b7c34fac14ab3bfe3c58386fcff04 at {username='slave'} at 127.0.99.133:34243: session id = f25b7c34fac14ab3bfe3c58386fcff04-6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:08.539947 2410 tablet_copy_source_session.cc:205] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:08.541594 4351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb7d8d7017f48be83670a06fc02b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:08.544162 4351 tablet_copy_client.cc:783] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:08.544324 4351 tablet_copy_client.cc:647] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:08.545729 4351 tablet_copy_client.cc:523] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:08.546897 4351 tablet_bootstrap.cc:492] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap starting.
I20240501 07:58:08.568962 4351 tablet_bootstrap.cc:492] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:08.569295 4351 tablet_bootstrap.cc:492] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Bootstrap complete.
I20240501 07:58:08.569463 4351 ts_tablet_manager.cc:1367] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240501 07:58:08.569945 4351 raft_consensus.cc:348] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:08.570266 4351 raft_consensus.cc:729] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Initialized, Role: FOLLOWER
I20240501 07:58:08.570384 4351 consensus_queue.cc:260] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:08.570742 4351 ts_tablet_manager.cc:1398] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:08.572221 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session f25b7c34fac14ab3bfe3c58386fcff04-6eb7d8d7017f48be83670a06fc02b26d received from {username='slave'} at 127.0.99.133:34243
I20240501 07:58:08.572305 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session f25b7c34fac14ab3bfe3c58386fcff04-6eb7d8d7017f48be83670a06fc02b26d on tablet 6eb7d8d7017f48be83670a06fc02b26d with peer f25b7c34fac14ab3bfe3c58386fcff04
I20240501 07:58:09.000795 3786 raft_consensus.cc:1203] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:09.001878 4361 raft_consensus.cc:2932] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:09.002277 3786 raft_consensus.cc:2932] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:09.003358 4361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32874 (request call id 0) took 1091 ms. Trace:
I20240501 07:58:09.003422 4361 rpcz_store.cc:276] 0501 07:58:07.911764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:07.911818 (+ 54us) service_pool.cc:225] Handling call
0501 07:58:07.912529 (+ 711us) log.cc:842] Serialized 257 byte log entry
0501 07:58:09.003344 (+1090815us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":54,"raft.queue_time_us":54,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":193,"wal-append.run_cpu_time_us":313,"wal-append.run_wall_time_us":1000379}
I20240501 07:58:09.004786 2240 catalog_manager.cc:5484] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 reported cstate change: config changed from index -1 to 4, VOTER f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:09.023519 2389 consensus_queue.cc:237] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
W20240501 07:58:09.024770 2326 consensus_peers.cc:485] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e -> Peer 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134:37621): Couldn't send request to peer 827b6164a742460a8c2086c5cb1222f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c35e3ea72543f1bd874e72f708cdf2. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:09.621361 4386 ts_tablet_manager.cc:902] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:09.621816 4386 tablet_copy_client.cc:308] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:09.622401 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet b6c35e3ea72543f1bd874e72f708cdf2 from peer 827b6164a742460a8c2086c5cb1222f7 ({username='slave'} at 127.0.99.134:35729)
I20240501 07:58:09.622501 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet b6c35e3ea72543f1bd874e72f708cdf2 from peer 827b6164a742460a8c2086c5cb1222f7 at {username='slave'} at 127.0.99.134:35729: session id = 827b6164a742460a8c2086c5cb1222f7-b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:09.623231 2410 tablet_copy_source_session.cc:205] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:09.624058 4386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c35e3ea72543f1bd874e72f708cdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:09.625738 4386 tablet_copy_client.cc:783] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:09.625890 4386 tablet_copy_client.cc:647] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:09.627036 4386 tablet_copy_client.cc:523] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:09.628022 4386 tablet_bootstrap.cc:492] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap starting.
I20240501 07:58:09.644790 4386 tablet_bootstrap.cc:492] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:09.645037 4386 tablet_bootstrap.cc:492] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Bootstrap complete.
I20240501 07:58:09.645206 4386 ts_tablet_manager.cc:1367] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20240501 07:58:09.645663 4386 raft_consensus.cc:348] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:09.645857 4386 raft_consensus.cc:729] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827b6164a742460a8c2086c5cb1222f7, State: Initialized, Role: FOLLOWER
I20240501 07:58:09.645946 4386 consensus_queue.cc:260] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:09.646236 4386 ts_tablet_manager.cc:1398] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 07:58:09.646811 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session 827b6164a742460a8c2086c5cb1222f7-b6c35e3ea72543f1bd874e72f708cdf2 received from {username='slave'} at 127.0.99.134:35729
I20240501 07:58:09.646907 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session 827b6164a742460a8c2086c5cb1222f7-b6c35e3ea72543f1bd874e72f708cdf2 on tablet b6c35e3ea72543f1bd874e72f708cdf2 with peer 827b6164a742460a8c2086c5cb1222f7
I20240501 07:58:10.166977 3926 raft_consensus.cc:1203] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:10.167948 4395 raft_consensus.cc:2932] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:10.168416 3926 raft_consensus.cc:2932] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:10.169441 4395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32892 (request call id 0) took 1146 ms. Trace:
I20240501 07:58:10.169636 4395 rpcz_store.cc:276] 0501 07:58:09.023200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:09.023253 (+ 53us) service_pool.cc:225] Handling call
0501 07:58:09.023882 (+ 629us) log.cc:842] Serialized 257 byte log entry
0501 07:58:10.169424 (+1145542us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":11648,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":131,"wal-append.run_cpu_time_us":361,"wal-append.run_wall_time_us":1000428}
I20240501 07:58:10.170794 2240 catalog_manager.cc:5484] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: config changed from index -1 to 4, VOTER 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:10.188331 2389 consensus_queue.cc:237] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
W20240501 07:58:10.192667 2325 consensus_peers.cc:485] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e -> Peer fe0605ab798147669ec83b685fc7ace0 (127.0.99.131:39669): Couldn't send request to peer fe0605ab798147669ec83b685fc7ace0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a0be484474f4cde809c8757a87c1bea. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:10.594826 4422 ts_tablet_manager.cc:902] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:10.595350 4422 tablet_copy_client.cc:308] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:10.598701 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet 6a0be484474f4cde809c8757a87c1bea from peer fe0605ab798147669ec83b685fc7ace0 ({username='slave'} at 127.0.99.131:44169)
I20240501 07:58:10.598803 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet 6a0be484474f4cde809c8757a87c1bea from peer fe0605ab798147669ec83b685fc7ace0 at {username='slave'} at 127.0.99.131:44169: session id = fe0605ab798147669ec83b685fc7ace0-6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:10.599598 2410 tablet_copy_source_session.cc:205] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:10.600431 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0be484474f4cde809c8757a87c1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:10.602126 4422 tablet_copy_client.cc:783] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:10.602290 4422 tablet_copy_client.cc:647] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:10.603406 4422 tablet_copy_client.cc:523] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:10.604332 4422 tablet_bootstrap.cc:492] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Bootstrap starting.
I20240501 07:58:10.621420 4422 tablet_bootstrap.cc:492] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:10.621691 4422 tablet_bootstrap.cc:492] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Bootstrap complete.
I20240501 07:58:10.621851 4422 ts_tablet_manager.cc:1367] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 07:58:10.622359 4422 raft_consensus.cc:348] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:10.622572 4422 raft_consensus.cc:729] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0605ab798147669ec83b685fc7ace0, State: Initialized, Role: FOLLOWER
I20240501 07:58:10.622668 4422 consensus_queue.cc:260] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:10.622985 4422 ts_tablet_manager.cc:1398] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:10.623617 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session fe0605ab798147669ec83b685fc7ace0-6a0be484474f4cde809c8757a87c1bea received from {username='slave'} at 127.0.99.131:44169
I20240501 07:58:10.623694 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session fe0605ab798147669ec83b685fc7ace0-6a0be484474f4cde809c8757a87c1bea on tablet 6a0be484474f4cde809c8757a87c1bea with peer fe0605ab798147669ec83b685fc7ace0
I20240501 07:58:11.133427 3523 raft_consensus.cc:1203] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:11.134459 4431 raft_consensus.cc:2932] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:11.134797 3523 raft_consensus.cc:2932] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:11.137455 2240 catalog_manager.cc:5484] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: config changed from index -1 to 4, VOTER fe0605ab798147669ec83b685fc7ace0 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:11.154735 2521 consensus_queue.cc:237] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
W20240501 07:58:11.158865 2455 consensus_peers.cc:485] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 -> Peer f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135:45899): Couldn't send request to peer f750f74c5d5b4295be7b3b3447f97797. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e734066a97d145338ab9c525d5d4c1dc. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:11.729410 4456 ts_tablet_manager.cc:902] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Initiating tablet copy from peer fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:11.729843 4456 tablet_copy_client.cc:308] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38583
I20240501 07:58:11.733039 2541 tablet_copy_service.cc:140] P fa963b55f7f04fdb9bd343e770f13dd5: Received BeginTabletCopySession request for tablet e734066a97d145338ab9c525d5d4c1dc from peer f750f74c5d5b4295be7b3b3447f97797 ({username='slave'} at 127.0.99.135:58509)
I20240501 07:58:11.733158 2541 tablet_copy_service.cc:161] P fa963b55f7f04fdb9bd343e770f13dd5: Beginning new tablet copy session on tablet e734066a97d145338ab9c525d5d4c1dc from peer f750f74c5d5b4295be7b3b3447f97797 at {username='slave'} at 127.0.99.135:58509: session id = f750f74c5d5b4295be7b3b3447f97797-e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:11.733879 2541 tablet_copy_source_session.cc:205] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:11.734681 4456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e734066a97d145338ab9c525d5d4c1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:11.736589 4456 tablet_copy_client.cc:783] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:11.736730 4456 tablet_copy_client.cc:647] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:11.737847 4456 tablet_copy_client.cc:523] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:11.738895 4456 tablet_bootstrap.cc:492] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Bootstrap starting.
I20240501 07:58:11.756986 4456 tablet_bootstrap.cc:492] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:11.757265 4456 tablet_bootstrap.cc:492] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Bootstrap complete.
I20240501 07:58:11.757462 4456 ts_tablet_manager.cc:1367] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20240501 07:58:11.757987 4456 raft_consensus.cc:348] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:11.758205 4456 raft_consensus.cc:729] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f750f74c5d5b4295be7b3b3447f97797, State: Initialized, Role: FOLLOWER
I20240501 07:58:11.758312 4456 consensus_queue.cc:260] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:11.758677 4456 ts_tablet_manager.cc:1398] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:11.759248 2541 tablet_copy_service.cc:342] P fa963b55f7f04fdb9bd343e770f13dd5: Request end of tablet copy session f750f74c5d5b4295be7b3b3447f97797-e734066a97d145338ab9c525d5d4c1dc received from {username='slave'} at 127.0.99.135:58509
I20240501 07:58:11.759326 2541 tablet_copy_service.cc:433] P fa963b55f7f04fdb9bd343e770f13dd5: ending tablet copy session f750f74c5d5b4295be7b3b3447f97797-e734066a97d145338ab9c525d5d4c1dc on tablet e734066a97d145338ab9c525d5d4c1dc with peer f750f74c5d5b4295be7b3b3447f97797
I20240501 07:58:12.160652 4057 raft_consensus.cc:1203] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:12.161578 4465 raft_consensus.cc:2932] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:12.162016 4057 raft_consensus.cc:2932] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:12.163120 4465 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40226 (request call id 0) took 1008 ms. Trace:
I20240501 07:58:12.163229 4465 rpcz_store.cc:276] 0501 07:58:11.154387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:11.154453 (+ 66us) service_pool.cc:225] Handling call
0501 07:58:11.155278 (+ 825us) log.cc:842] Serialized 257 byte log entry
0501 07:58:12.163101 (+1007823us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":76,"raft.queue_time_us":33,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":145,"wal-append.run_cpu_time_us":325,"wal-append.run_wall_time_us":1000395}
I20240501 07:58:12.165218 2240 catalog_manager.cc:5484] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: config changed from index -1 to 4, VOTER f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:12.182788 2521 consensus_queue.cc:237] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
W20240501 07:58:12.184280 2457 consensus_peers.cc:485] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 -> Peer 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132:35435): Couldn't send request to peer 3199fb0a9d4d448687cd049ae66601c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b4663e0650c4209bbb88e87291af937. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:12.785390 4489 ts_tablet_manager.cc:902] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Initiating tablet copy from peer fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:12.785887 4489 tablet_copy_client.cc:308] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38583
I20240501 07:58:12.786433 2541 tablet_copy_service.cc:140] P fa963b55f7f04fdb9bd343e770f13dd5: Received BeginTabletCopySession request for tablet 8b4663e0650c4209bbb88e87291af937 from peer 3199fb0a9d4d448687cd049ae66601c9 ({username='slave'} at 127.0.99.132:35185)
I20240501 07:58:12.786550 2541 tablet_copy_service.cc:161] P fa963b55f7f04fdb9bd343e770f13dd5: Beginning new tablet copy session on tablet 8b4663e0650c4209bbb88e87291af937 from peer 3199fb0a9d4d448687cd049ae66601c9 at {username='slave'} at 127.0.99.132:35185: session id = 3199fb0a9d4d448687cd049ae66601c9-8b4663e0650c4209bbb88e87291af937
I20240501 07:58:12.787348 2541 tablet_copy_source_session.cc:205] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:12.788204 4489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4663e0650c4209bbb88e87291af937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:12.790171 4489 tablet_copy_client.cc:783] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:12.790367 4489 tablet_copy_client.cc:647] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:12.791611 4489 tablet_copy_client.cc:523] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:12.792616 4489 tablet_bootstrap.cc:492] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap starting.
I20240501 07:58:12.811551 4489 tablet_bootstrap.cc:492] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:12.811834 4489 tablet_bootstrap.cc:492] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Bootstrap complete.
I20240501 07:58:12.811980 4489 ts_tablet_manager.cc:1367] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240501 07:58:12.812466 4489 raft_consensus.cc:348] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:12.812667 4489 raft_consensus.cc:729] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3199fb0a9d4d448687cd049ae66601c9, State: Initialized, Role: FOLLOWER
I20240501 07:58:12.812742 4489 consensus_queue.cc:260] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:12.813056 4489 ts_tablet_manager.cc:1398] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:12.813670 2541 tablet_copy_service.cc:342] P fa963b55f7f04fdb9bd343e770f13dd5: Request end of tablet copy session 3199fb0a9d4d448687cd049ae66601c9-8b4663e0650c4209bbb88e87291af937 received from {username='slave'} at 127.0.99.132:35185
I20240501 07:58:12.813766 2541 tablet_copy_service.cc:433] P fa963b55f7f04fdb9bd343e770f13dd5: ending tablet copy session 3199fb0a9d4d448687cd049ae66601c9-8b4663e0650c4209bbb88e87291af937 on tablet 8b4663e0650c4209bbb88e87291af937 with peer 3199fb0a9d4d448687cd049ae66601c9
I20240501 07:58:13.379056 3654 raft_consensus.cc:1203] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:13.379995 4497 raft_consensus.cc:2932] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:13.380388 3654 raft_consensus.cc:2932] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:13.381424 4497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40240 (request call id 0) took 1199 ms. Trace:
I20240501 07:58:13.381518 4497 rpcz_store.cc:276] 0501 07:58:12.182445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:12.182514 (+ 69us) service_pool.cc:225] Handling call
0501 07:58:12.183195 (+ 681us) log.cc:842] Serialized 257 byte log entry
0501 07:58:13.381407 (+1198212us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":110,"thread_start_us":108,"threads_started":1,"wal-append.queue_time_us":164,"wal-append.run_cpu_time_us":290,"wal-append.run_wall_time_us":1000349}
I20240501 07:58:13.383448 2247 catalog_manager.cc:5484] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: config changed from index -1 to 4, VOTER 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:13.443195 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.445309 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.448295 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.455294 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.456655 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.463287 2501 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:13.466288 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.498482 2521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e734066a97d145338ab9c525d5d4c1dc"
dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40280
I20240501 07:58:13.498652 2521 raft_consensus.cc:595] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:13.547498 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.547786 4037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.554241 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.560637 2501 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:13.563133 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.576108 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.576517 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.599059 4465 raft_consensus.cc:979] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: : Instructing follower f750f74c5d5b4295be7b3b3447f97797 to start an election
I20240501 07:58:13.599201 4465 raft_consensus.cc:1067] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Signalling peer f750f74c5d5b4295be7b3b3447f97797 to start an election
I20240501 07:58:13.599788 4057 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e734066a97d145338ab9c525d5d4c1dc"
dest_uuid: "f750f74c5d5b4295be7b3b3447f97797"
from {username='slave'} at 127.0.99.130:41837
I20240501 07:58:13.599964 4057 raft_consensus.cc:482] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:13.600081 4057 raft_consensus.cc:3037] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:13.600922 4057 raft_consensus.cc:504] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:13.601747 4057 leader_election.cc:288] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [CANDIDATE]: Term 2 election: Requested vote from peers fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:13.602141 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e734066a97d145338ab9c525d5d4c1dc" candidate_uuid: "f750f74c5d5b4295be7b3b3447f97797" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
I20240501 07:58:13.602322 2521 raft_consensus.cc:3032] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:13.602375 2521 raft_consensus.cc:729] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:13.602550 2521 consensus_queue.cc:260] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:13.602784 2521 raft_consensus.cc:3037] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:13.603879 2521 raft_consensus.cc:2454] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f750f74c5d5b4295be7b3b3447f97797 in term 2.
I20240501 07:58:13.604208 3992 leader_election.cc:302] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f750f74c5d5b4295be7b3b3447f97797, fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:13.605517 4586 raft_consensus.cc:2781] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:13.605708 4586 raft_consensus.cc:686] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 2 LEADER]: Becoming Leader. State: Replica: f750f74c5d5b4295be7b3b3447f97797, State: Running, Role: LEADER
I20240501 07:58:13.606366 4586 consensus_queue.cc:237] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:13.608726 2240 catalog_manager.cc:5484] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: term changed from 1 to 2, leader changed from fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) to f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "f750f74c5d5b4295be7b3b3447f97797" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:13.662575 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.663192 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.664959 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.666661 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.670075 2501 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:13.671401 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.691797 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.712599 2521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b4663e0650c4209bbb88e87291af937"
dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40316
I20240501 07:58:13.712734 2521 raft_consensus.cc:595] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:13.766705 3634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.768150 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.768844 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.787663 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.789834 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.790114 2501 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:13.790113 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.803411 4497 raft_consensus.cc:979] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: : Instructing follower 3199fb0a9d4d448687cd049ae66601c9 to start an election
I20240501 07:58:13.803506 4497 raft_consensus.cc:1067] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Signalling peer 3199fb0a9d4d448687cd049ae66601c9 to start an election
I20240501 07:58:13.804039 3654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b4663e0650c4209bbb88e87291af937"
dest_uuid: "3199fb0a9d4d448687cd049ae66601c9"
from {username='slave'} at 127.0.99.130:37753
I20240501 07:58:13.804239 3654 raft_consensus.cc:482] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:13.804333 3654 raft_consensus.cc:3037] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:13.805406 3654 raft_consensus.cc:504] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:13.806389 3654 leader_election.cc:288] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [CANDIDATE]: Term 2 election: Requested vote from peers fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:13.806807 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4663e0650c4209bbb88e87291af937" candidate_uuid: "3199fb0a9d4d448687cd049ae66601c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
I20240501 07:58:13.806916 2521 raft_consensus.cc:3032] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:13.806952 2521 raft_consensus.cc:729] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:13.807196 2521 consensus_queue.cc:260] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:13.807296 2521 raft_consensus.cc:3037] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:13.808205 2521 raft_consensus.cc:2454] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3199fb0a9d4d448687cd049ae66601c9 in term 2.
I20240501 07:58:13.808607 3589 leader_election.cc:302] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3199fb0a9d4d448687cd049ae66601c9, fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:13.808931 4496 raft_consensus.cc:2781] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:13.809113 4496 raft_consensus.cc:686] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 2 LEADER]: Becoming Leader. State: Replica: 3199fb0a9d4d448687cd049ae66601c9, State: Running, Role: LEADER
I20240501 07:58:13.809439 4496 consensus_queue.cc:237] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:13.811919 2241 catalog_manager.cc:5484] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: term changed from 1 to 2, leader changed from fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) to 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "3199fb0a9d4d448687cd049ae66601c9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:13.883088 3634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.883813 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.886236 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.887722 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.888689 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.888896 2501 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:13.898079 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.931988 2521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96af3865d3ef469ea5b24cba20519bb3"
dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40364
I20240501 07:58:13.932130 2521 raft_consensus.cc:595] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:13.982769 3634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.983064 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:13.983160 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:13.988528 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:13.997591 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.006023 2501 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:14.021155 3503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.023547 4465 raft_consensus.cc:979] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: : Instructing follower fe0605ab798147669ec83b685fc7ace0 to start an election
I20240501 07:58:14.023635 4465 raft_consensus.cc:1067] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Signalling peer fe0605ab798147669ec83b685fc7ace0 to start an election
I20240501 07:58:14.024183 3523 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96af3865d3ef469ea5b24cba20519bb3"
dest_uuid: "fe0605ab798147669ec83b685fc7ace0"
from {username='slave'} at 127.0.99.130:37887
I20240501 07:58:14.024372 3523 raft_consensus.cc:482] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:14.024471 3523 raft_consensus.cc:3037] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.025532 3523 raft_consensus.cc:504] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:14.026510 3523 leader_election.cc:288] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [CANDIDATE]: Term 2 election: Requested vote from peers fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:14.026873 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96af3865d3ef469ea5b24cba20519bb3" candidate_uuid: "fe0605ab798147669ec83b685fc7ace0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
I20240501 07:58:14.027020 2521 raft_consensus.cc:3032] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:14.027066 2521 raft_consensus.cc:729] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:14.027204 2521 consensus_queue.cc:260] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:14.027331 2521 raft_consensus.cc:3037] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.028231 2521 raft_consensus.cc:2454] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0605ab798147669ec83b685fc7ace0 in term 2.
I20240501 07:58:14.028540 3458 leader_election.cc:302] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: fa963b55f7f04fdb9bd343e770f13dd5, fe0605ab798147669ec83b685fc7ace0; no voters:
I20240501 07:58:14.032999 4748 raft_consensus.cc:2781] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:14.033174 4748 raft_consensus.cc:686] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 2 LEADER]: Becoming Leader. State: Replica: fe0605ab798147669ec83b685fc7ace0, State: Running, Role: LEADER
I20240501 07:58:14.033491 4748 consensus_queue.cc:237] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:14.035851 2241 catalog_manager.cc:5484] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 reported cstate change: term changed from 1 to 2, leader changed from fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) to fe0605ab798147669ec83b685fc7ace0 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fe0605ab798147669ec83b685fc7ace0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:14.104458 3634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.105180 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.105302 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.107573 2501 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:14.120966 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.121290 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.122484 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.159147 2521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c7c7acd9cd91461197fc2e2c86bd570f"
dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40388
I20240501 07:58:14.159287 2521 raft_consensus.cc:595] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:14.202343 2521 raft_consensus.cc:1261] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Refusing update from remote peer f750f74c5d5b4295be7b3b3447f97797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:14.206499 4825 consensus_queue.cc:1035] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:14.213152 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.215088 3634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.217634 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.219177 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.219317 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.225940 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.231209 2501 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:14.262622 2521 raft_consensus.cc:1261] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Refusing update from remote peer 3199fb0a9d4d448687cd049ae66601c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:14.263257 4663 consensus_queue.cc:1035] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:14.319190 3634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.319358 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.319741 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.323612 2501 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:14.332181 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.334528 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.336212 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.362834 4465 raft_consensus.cc:979] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: : Instructing follower 3199fb0a9d4d448687cd049ae66601c9 to start an election
I20240501 07:58:14.362957 4465 raft_consensus.cc:1067] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Signalling peer 3199fb0a9d4d448687cd049ae66601c9 to start an election
I20240501 07:58:14.363478 3654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7c7acd9cd91461197fc2e2c86bd570f"
dest_uuid: "3199fb0a9d4d448687cd049ae66601c9"
from {username='slave'} at 127.0.99.130:37753
I20240501 07:58:14.363629 3654 raft_consensus.cc:482] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:14.363688 3654 raft_consensus.cc:3037] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.364751 3654 raft_consensus.cc:504] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:14.365067 3654 leader_election.cc:288] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [CANDIDATE]: Term 2 election: Requested vote from peers fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:14.365391 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c7acd9cd91461197fc2e2c86bd570f" candidate_uuid: "3199fb0a9d4d448687cd049ae66601c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
I20240501 07:58:14.365535 2521 raft_consensus.cc:3032] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:14.365581 2521 raft_consensus.cc:729] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:14.365723 2521 consensus_queue.cc:260] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:14.365865 2521 raft_consensus.cc:3037] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.366763 2521 raft_consensus.cc:2454] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3199fb0a9d4d448687cd049ae66601c9 in term 2.
I20240501 07:58:14.367063 3589 leader_election.cc:302] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3199fb0a9d4d448687cd049ae66601c9, fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:14.367233 4496 raft_consensus.cc:2781] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:14.367354 4496 raft_consensus.cc:686] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 2 LEADER]: Becoming Leader. State: Replica: 3199fb0a9d4d448687cd049ae66601c9, State: Running, Role: LEADER
I20240501 07:58:14.367478 4496 consensus_queue.cc:237] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:14.369037 2241 catalog_manager.cc:5484] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: term changed from 1 to 2, leader changed from fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) to 3199fb0a9d4d448687cd049ae66601c9 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "3199fb0a9d4d448687cd049ae66601c9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:14.376149 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "914df5ce34d5464a9a3876fb0cc472f1"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33012
I20240501 07:58:14.376302 2389 raft_consensus.cc:595] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:14.432502 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.433986 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.434671 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.439745 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.447678 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.450630 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:14.451169 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.541713 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.542256 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.543344 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.545477 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.552179 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:14.554337 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.556317 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.602705 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a0be484474f4cde809c8757a87c1bea"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33062
I20240501 07:58:14.602834 2389 raft_consensus.cc:595] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:14.634949 2521 raft_consensus.cc:1261] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Refusing update from remote peer fe0605ab798147669ec83b685fc7ace0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:14.635599 4975 consensus_queue.cc:1035] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:14.655146 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.656383 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.664785 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.665150 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:14.666167 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.677402 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.678102 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.763211 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.763525 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.765492 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.767762 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:14.768103 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.784194 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.785941 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.814837 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eff829f74ea44c7c9cefa72260ac2165"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39854
I20240501 07:58:14.814980 2389 raft_consensus.cc:595] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:14.866417 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.867707 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.868218 2369 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:14.872005 4037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.880548 3906 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:14.882002 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.882336 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:14.901803 4395 raft_consensus.cc:979] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower f750f74c5d5b4295be7b3b3447f97797 to start an election
I20240501 07:58:14.901901 4395 raft_consensus.cc:1067] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer f750f74c5d5b4295be7b3b3447f97797 to start an election
I20240501 07:58:14.902487 4056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff829f74ea44c7c9cefa72260ac2165"
dest_uuid: "f750f74c5d5b4295be7b3b3447f97797"
from {username='slave'} at 127.0.99.129:39953
I20240501 07:58:14.902614 4056 raft_consensus.cc:482] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:14.902666 4056 raft_consensus.cc:3037] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.903751 4056 raft_consensus.cc:504] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:14.904150 4056 leader_election.cc:288] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:14.904911 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff829f74ea44c7c9cefa72260ac2165" candidate_uuid: "f750f74c5d5b4295be7b3b3447f97797" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:14.905097 2389 raft_consensus.cc:3032] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:14.905155 2389 raft_consensus.cc:729] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:14.905294 2389 consensus_queue.cc:260] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:14.905552 2389 raft_consensus.cc:3037] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.906481 2389 raft_consensus.cc:2454] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f750f74c5d5b4295be7b3b3447f97797 in term 2.
I20240501 07:58:14.906807 3992 leader_election.cc:302] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e, f750f74c5d5b4295be7b3b3447f97797; no voters:
I20240501 07:58:14.907104 4837 raft_consensus.cc:2781] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:14.907245 4837 raft_consensus.cc:686] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 2 LEADER]: Becoming Leader. State: Replica: f750f74c5d5b4295be7b3b3447f97797, State: Running, Role: LEADER
I20240501 07:58:14.907362 4837 consensus_queue.cc:237] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:14.908950 2241 catalog_manager.cc:5484] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to f750f74c5d5b4295be7b3b3447f97797 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "f750f74c5d5b4295be7b3b3447f97797" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:14.911175 2521 raft_consensus.cc:1261] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Refusing update from remote peer 3199fb0a9d4d448687cd049ae66601c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:14.911883 4496 consensus_queue.cc:1035] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:14.919502 4395 raft_consensus.cc:979] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower 827b6164a742460a8c2086c5cb1222f7 to start an election
I20240501 07:58:14.919675 5068 raft_consensus.cc:1067] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer 827b6164a742460a8c2086c5cb1222f7 to start an election
I20240501 07:58:14.920171 3926 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "914df5ce34d5464a9a3876fb0cc472f1"
dest_uuid: "827b6164a742460a8c2086c5cb1222f7"
from {username='slave'} at 127.0.99.129:53211
I20240501 07:58:14.920358 3926 raft_consensus.cc:482] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:14.920475 3926 raft_consensus.cc:3037] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.921319 3926 raft_consensus.cc:504] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:14.922116 3926 leader_election.cc:288] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:14.922469 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "914df5ce34d5464a9a3876fb0cc472f1" candidate_uuid: "827b6164a742460a8c2086c5cb1222f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:14.922572 2389 raft_consensus.cc:3032] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:14.922626 2389 raft_consensus.cc:729] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:14.922802 2389 consensus_queue.cc:260] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:14.922955 2389 raft_consensus.cc:3037] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:14.923774 2389 raft_consensus.cc:2454] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827b6164a742460a8c2086c5cb1222f7 in term 2.
I20240501 07:58:14.924137 3861 leader_election.cc:302] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 827b6164a742460a8c2086c5cb1222f7, bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:14.924477 5087 raft_consensus.cc:2781] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:14.924697 5087 raft_consensus.cc:686] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 2 LEADER]: Becoming Leader. State: Replica: 827b6164a742460a8c2086c5cb1222f7, State: Running, Role: LEADER
I20240501 07:58:14.925035 5087 consensus_queue.cc:237] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:14.927433 2247 catalog_manager.cc:5484] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "827b6164a742460a8c2086c5cb1222f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:14.983992 3906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:14.984177 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.985216 2369 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:14.985975 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:14.987283 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.011683 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.018705 3503 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.041680 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6c35e3ea72543f1bd874e72f708cdf2"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39888
I20240501 07:58:15.041817 2389 raft_consensus.cc:595] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:15.093878 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.094470 2369 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:15.094826 3906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.096333 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.099267 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.112377 5068 raft_consensus.cc:979] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower fe0605ab798147669ec83b685fc7ace0 to start an election
I20240501 07:58:15.112479 5068 raft_consensus.cc:1067] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer fe0605ab798147669ec83b685fc7ace0 to start an election
I20240501 07:58:15.112874 3523 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a0be484474f4cde809c8757a87c1bea"
dest_uuid: "fe0605ab798147669ec83b685fc7ace0"
from {username='slave'} at 127.0.99.129:44065
I20240501 07:58:15.113003 3523 raft_consensus.cc:482] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:15.113054 3523 raft_consensus.cc:3037] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.114130 3523 raft_consensus.cc:504] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:15.114507 3523 leader_election.cc:288] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:15.115128 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0be484474f4cde809c8757a87c1bea" candidate_uuid: "fe0605ab798147669ec83b685fc7ace0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:15.115245 2389 raft_consensus.cc:3032] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:15.115280 2389 raft_consensus.cc:729] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:15.115404 2389 consensus_queue.cc:260] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:15.115522 2389 raft_consensus.cc:3037] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.116423 2389 raft_consensus.cc:2454] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0605ab798147669ec83b685fc7ace0 in term 2.
I20240501 07:58:15.116717 3458 leader_election.cc:302] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e, fe0605ab798147669ec83b685fc7ace0; no voters:
I20240501 07:58:15.116927 4982 raft_consensus.cc:2781] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:15.117007 4982 raft_consensus.cc:686] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 2 LEADER]: Becoming Leader. State: Replica: fe0605ab798147669ec83b685fc7ace0, State: Running, Role: LEADER
I20240501 07:58:15.117118 4982 consensus_queue.cc:237] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:15.118623 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.118829 2244 catalog_manager.cc:5484] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to fe0605ab798147669ec83b685fc7ace0 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fe0605ab798147669ec83b685fc7ace0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:15.125811 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.203892 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.207832 2369 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:15.212054 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.218055 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.218314 3906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.225735 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.227747 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.258352 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eb7d8d7017f48be83670a06fc02b26d"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39918
I20240501 07:58:15.258495 2389 raft_consensus.cc:595] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:15.293548 5068 raft_consensus.cc:979] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower 827b6164a742460a8c2086c5cb1222f7 to start an election
I20240501 07:58:15.293666 5068 raft_consensus.cc:1067] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer 827b6164a742460a8c2086c5cb1222f7 to start an election
I20240501 07:58:15.294102 3926 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6c35e3ea72543f1bd874e72f708cdf2"
dest_uuid: "827b6164a742460a8c2086c5cb1222f7"
from {username='slave'} at 127.0.99.129:53211
I20240501 07:58:15.294238 3926 raft_consensus.cc:482] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:15.294324 3926 raft_consensus.cc:3037] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.295447 3926 raft_consensus.cc:504] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:15.295770 3926 leader_election.cc:288] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:15.296211 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c35e3ea72543f1bd874e72f708cdf2" candidate_uuid: "827b6164a742460a8c2086c5cb1222f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:15.296346 2389 raft_consensus.cc:3032] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:15.296399 2389 raft_consensus.cc:729] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:15.296538 2389 consensus_queue.cc:260] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:15.296638 2389 raft_consensus.cc:3037] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.297641 2389 raft_consensus.cc:2454] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827b6164a742460a8c2086c5cb1222f7 in term 2.
I20240501 07:58:15.297992 3861 leader_election.cc:302] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 827b6164a742460a8c2086c5cb1222f7, bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:15.298205 5087 raft_consensus.cc:2781] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:15.298295 5087 raft_consensus.cc:686] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 2 LEADER]: Becoming Leader. State: Replica: 827b6164a742460a8c2086c5cb1222f7, State: Running, Role: LEADER
I20240501 07:58:15.298422 5087 consensus_queue.cc:237] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:15.300087 2241 catalog_manager.cc:5484] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to 827b6164a742460a8c2086c5cb1222f7 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "827b6164a742460a8c2086c5cb1222f7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:15.312886 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.313553 2369 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:15.316586 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.317379 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.318646 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.330457 3758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:15.353277 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.427596 2389 raft_consensus.cc:1261] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer f750f74c5d5b4295be7b3b3447f97797: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:15.428193 5069 consensus_queue.cc:1035] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:15.454056 2389 raft_consensus.cc:1261] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer 827b6164a742460a8c2086c5cb1222f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:15.454603 5088 consensus_queue.cc:1035] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:15.617105 4395 raft_consensus.cc:979] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower f25b7c34fac14ab3bfe3c58386fcff04 to start an election
I20240501 07:58:15.617236 4395 raft_consensus.cc:1067] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer f25b7c34fac14ab3bfe3c58386fcff04 to start an election
I20240501 07:58:15.617708 3786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6eb7d8d7017f48be83670a06fc02b26d"
dest_uuid: "f25b7c34fac14ab3bfe3c58386fcff04"
from {username='slave'} at 127.0.99.129:60125
I20240501 07:58:15.617913 3786 raft_consensus.cc:482] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:15.617991 3786 raft_consensus.cc:3037] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.618975 3786 raft_consensus.cc:504] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:15.619750 3786 leader_election.cc:288] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:15.620111 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb7d8d7017f48be83670a06fc02b26d" candidate_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:15.620234 2389 raft_consensus.cc:3032] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:15.620277 2389 raft_consensus.cc:729] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:15.620422 2389 consensus_queue.cc:260] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:15.620518 2389 raft_consensus.cc:3037] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:15.621529 2389 raft_consensus.cc:2454] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f25b7c34fac14ab3bfe3c58386fcff04 in term 2.
I20240501 07:58:15.621851 3720 leader_election.cc:302] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf25fbb990fb46ccb496debdcc7cf91e, f25b7c34fac14ab3bfe3c58386fcff04; no voters:
I20240501 07:58:15.622180 5274 raft_consensus.cc:2781] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:15.622392 5274 raft_consensus.cc:686] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 LEADER]: Becoming Leader. State: Replica: f25b7c34fac14ab3bfe3c58386fcff04, State: Running, Role: LEADER
I20240501 07:58:15.622601 5274 consensus_queue.cc:237] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:15.624469 2240 catalog_manager.cc:5484] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to f25b7c34fac14ab3bfe3c58386fcff04 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:15.637670 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.638088 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.638321 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.640234 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.642128 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.650830 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.670715 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.705727 4056 consensus_queue.cc:237] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:15.706555 5242 raft_consensus.cc:2932] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:15.709020 2240 catalog_manager.cc:5484] T e734066a97d145338ab9c525d5d4c1dc P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "f750f74c5d5b4295be7b3b3447f97797" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:15.711905 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet e734066a97d145338ab9c525d5d4c1dc with delete_type TABLET_DATA_TOMBSTONED (TS fa963b55f7f04fdb9bd343e770f13dd5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33458
I20240501 07:58:15.712368 5308 tablet_replica.cc:330] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: stopping tablet replica
I20240501 07:58:15.712646 5308 raft_consensus.cc:2229] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:15.712814 5308 raft_consensus.cc:2258] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:15.713654 5308 ts_tablet_manager.cc:1875] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:15.716090 5308 ts_tablet_manager.cc:1888] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:15.716184 5308 log.cc:1196] T e734066a97d145338ab9c525d5d4c1dc P fa963b55f7f04fdb9bd343e770f13dd5: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:15.716768 2234 catalog_manager.cc:4830] TS fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583): tablet e734066a97d145338ab9c525d5d4c1dc (table rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]) successfully deleted
I20240501 07:58:15.741184 2389 raft_consensus.cc:1261] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer fe0605ab798147669ec83b685fc7ace0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:15.741739 4975 consensus_queue.cc:1035] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:15.771457 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.777011 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.779901 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.780169 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.781487 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.782563 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.786988 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.841382 3654 consensus_queue.cc:237] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:15.842133 4496 raft_consensus.cc:2932] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:15.844489 2246 catalog_manager.cc:5484] T 8b4663e0650c4209bbb88e87291af937 P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "3199fb0a9d4d448687cd049ae66601c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:15.846858 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b4663e0650c4209bbb88e87291af937 with delete_type TABLET_DATA_TOMBSTONED (TS fa963b55f7f04fdb9bd343e770f13dd5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33458
I20240501 07:58:15.847139 5308 tablet_replica.cc:330] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: stopping tablet replica
I20240501 07:58:15.847309 5308 raft_consensus.cc:2229] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:15.847431 5308 raft_consensus.cc:2258] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:15.848188 5308 ts_tablet_manager.cc:1875] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:15.850225 5308 ts_tablet_manager.cc:1888] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:15.850306 5308 log.cc:1196] T 8b4663e0650c4209bbb88e87291af937 P fa963b55f7f04fdb9bd343e770f13dd5: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8b4663e0650c4209bbb88e87291af937
I20240501 07:58:15.850837 2234 catalog_manager.cc:4830] TS fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583): tablet 8b4663e0650c4209bbb88e87291af937 (table rebalance_test_table_0 [id=4119ed7f60404f07b49c27fc5727c0be]) successfully deleted
I20240501 07:58:15.873876 2389 raft_consensus.cc:1261] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer 827b6164a742460a8c2086c5cb1222f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:15.874311 5087 consensus_queue.cc:1035] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:15.901177 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.904387 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.904460 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.909183 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.918040 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:15.925357 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.931603 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:15.969882 3523 consensus_queue.cc:237] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:15.970656 5321 raft_consensus.cc:2932] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:15.973304 2240 catalog_manager.cc:5484] T 96af3865d3ef469ea5b24cba20519bb3 P fe0605ab798147669ec83b685fc7ace0 reported cstate change: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "fe0605ab798147669ec83b685fc7ace0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:15.976001 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 96af3865d3ef469ea5b24cba20519bb3 with delete_type TABLET_DATA_TOMBSTONED (TS fa963b55f7f04fdb9bd343e770f13dd5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33458
I20240501 07:58:15.976218 5308 tablet_replica.cc:330] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: stopping tablet replica
I20240501 07:58:15.976342 5308 raft_consensus.cc:2229] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:15.976447 5308 raft_consensus.cc:2258] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:15.976884 5308 ts_tablet_manager.cc:1875] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:15.978746 5308 ts_tablet_manager.cc:1888] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:15.978832 5308 log.cc:1196] T 96af3865d3ef469ea5b24cba20519bb3 P fa963b55f7f04fdb9bd343e770f13dd5: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:15.979370 2234 catalog_manager.cc:4830] TS fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583): tablet 96af3865d3ef469ea5b24cba20519bb3 (table rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]) successfully deleted
I20240501 07:58:16.031510 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.035734 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.036118 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.036834 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.043571 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.045091 2389 raft_consensus.cc:1261] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer f25b7c34fac14ab3bfe3c58386fcff04: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:16.045579 5275 consensus_queue.cc:1035] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:16.055191 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.057371 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.098488 3654 consensus_queue.cc:237] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } }
I20240501 07:58:16.099284 5360 raft_consensus.cc:2932] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:16.101431 2240 catalog_manager.cc:5484] T c7c7acd9cd91461197fc2e2c86bd570f P 3199fb0a9d4d448687cd049ae66601c9 reported cstate change: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "3199fb0a9d4d448687cd049ae66601c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3199fb0a9d4d448687cd049ae66601c9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 35435 } } }
I20240501 07:58:16.104079 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c7acd9cd91461197fc2e2c86bd570f with delete_type TABLET_DATA_TOMBSTONED (TS fa963b55f7f04fdb9bd343e770f13dd5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33458
I20240501 07:58:16.104285 5308 tablet_replica.cc:330] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: stopping tablet replica
I20240501 07:58:16.104403 5308 raft_consensus.cc:2229] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.104506 5308 raft_consensus.cc:2258] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.104961 5308 ts_tablet_manager.cc:1875] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.106772 5308 ts_tablet_manager.cc:1888] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.106851 5308 log.cc:1196] T c7c7acd9cd91461197fc2e2c86bd570f P fa963b55f7f04fdb9bd343e770f13dd5: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:16.107373 2234 catalog_manager.cc:4830] TS fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583): tablet c7c7acd9cd91461197fc2e2c86bd570f (table rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]) successfully deleted
I20240501 07:58:16.159132 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.161499 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.162739 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.166852 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.167698 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.178159 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.183286 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.221052 3926 consensus_queue.cc:237] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:16.221626 5088 raft_consensus.cc:2932] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:16.223829 2241 catalog_manager.cc:5484] T 914df5ce34d5464a9a3876fb0cc472f1 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "827b6164a742460a8c2086c5cb1222f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:16.226727 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet 914df5ce34d5464a9a3876fb0cc472f1 with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:16.227200 5527 tablet_replica.cc:330] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:16.227463 5527 raft_consensus.cc:2229] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.227612 5527 raft_consensus.cc:2258] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.228842 5527 ts_tablet_manager.cc:1875] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.231377 5527 ts_tablet_manager.cc:1888] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.231470 5527 log.cc:1196] T 914df5ce34d5464a9a3876fb0cc472f1 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:16.232038 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet 914df5ce34d5464a9a3876fb0cc472f1 (table rebalance_test_table_0 [id=4119ed7f60404f07b49c27fc5727c0be]) successfully deleted
I20240501 07:58:16.279079 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.279348 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.280059 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.283138 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.292994 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.293506 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.303520 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.341720 3523 consensus_queue.cc:237] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } }
I20240501 07:58:16.342308 5321 raft_consensus.cc:2932] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:16.344909 2240 catalog_manager.cc:5484] T 6a0be484474f4cde809c8757a87c1bea P fe0605ab798147669ec83b685fc7ace0 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "fe0605ab798147669ec83b685fc7ace0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe0605ab798147669ec83b685fc7ace0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39669 } } }
I20240501 07:58:16.347354 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a0be484474f4cde809c8757a87c1bea with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:16.347580 5527 tablet_replica.cc:330] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:16.347729 5527 raft_consensus.cc:2229] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.347834 5527 raft_consensus.cc:2258] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.348497 5527 ts_tablet_manager.cc:1875] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.350359 5527 ts_tablet_manager.cc:1888] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.350430 5527 log.cc:1196] T 6a0be484474f4cde809c8757a87c1bea P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:16.350950 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet 6a0be484474f4cde809c8757a87c1bea (table rebalance_test_table_4 [id=d541bdfe4e0042b8b8361aa1e34290d7]) successfully deleted
I20240501 07:58:16.405514 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.408056 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.413908 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.418361 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.425460 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.432098 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.437310 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.477622 4056 consensus_queue.cc:237] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } }
I20240501 07:58:16.478327 5242 raft_consensus.cc:2932] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:16.480556 2246 catalog_manager.cc:5484] T eff829f74ea44c7c9cefa72260ac2165 P f750f74c5d5b4295be7b3b3447f97797 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "f750f74c5d5b4295be7b3b3447f97797" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f750f74c5d5b4295be7b3b3447f97797" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45899 } } }
I20240501 07:58:16.483067 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet eff829f74ea44c7c9cefa72260ac2165 with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:16.483350 5527 tablet_replica.cc:330] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:16.483501 5527 raft_consensus.cc:2229] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.483623 5527 raft_consensus.cc:2258] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.484093 5527 ts_tablet_manager.cc:1875] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.486287 5527 ts_tablet_manager.cc:1888] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.486362 5527 log.cc:1196] T eff829f74ea44c7c9cefa72260ac2165 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:16.486876 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet eff829f74ea44c7c9cefa72260ac2165 (table rebalance_test_table_4 [id=d541bdfe4e0042b8b8361aa1e34290d7]) successfully deleted
I20240501 07:58:16.536206 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.537703 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.541582 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.543156 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.548074 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.553369 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.560977 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.606524 3926 consensus_queue.cc:237] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } }
I20240501 07:58:16.607086 5088 raft_consensus.cc:2932] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:16.609313 2244 catalog_manager.cc:5484] T b6c35e3ea72543f1bd874e72f708cdf2 P 827b6164a742460a8c2086c5cb1222f7 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "827b6164a742460a8c2086c5cb1222f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "827b6164a742460a8c2086c5cb1222f7" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37621 } } }
I20240501 07:58:16.611831 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c35e3ea72543f1bd874e72f708cdf2 with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:16.612062 5527 tablet_replica.cc:330] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:16.612214 5527 raft_consensus.cc:2229] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.612334 5527 raft_consensus.cc:2258] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.612968 5527 ts_tablet_manager.cc:1875] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.614786 5527 ts_tablet_manager.cc:1888] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.614867 5527 log.cc:1196] T b6c35e3ea72543f1bd874e72f708cdf2 P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:16.615414 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet b6c35e3ea72543f1bd874e72f708cdf2 (table rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]) successfully deleted
I20240501 07:58:16.665352 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.667613 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.667749 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.672956 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:16.673630 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.675516 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.683249 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:16.729262 3786 consensus_queue.cc:237] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } }
I20240501 07:58:16.729893 5274 raft_consensus.cc:2932] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:16.732100 2240 catalog_manager.cc:5484] T 6eb7d8d7017f48be83670a06fc02b26d P f25b7c34fac14ab3bfe3c58386fcff04 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f25b7c34fac14ab3bfe3c58386fcff04" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35243 } } }
I20240501 07:58:16.734423 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb7d8d7017f48be83670a06fc02b26d with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:16.734658 5527 tablet_replica.cc:330] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:16.734802 5527 raft_consensus.cc:2229] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:16.734908 5527 raft_consensus.cc:2258] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:16.735569 5527 ts_tablet_manager.cc:1875] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:16.737182 5527 ts_tablet_manager.cc:1888] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:16.737242 5527 log.cc:1196] T 6eb7d8d7017f48be83670a06fc02b26d P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:16.737694 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet 6eb7d8d7017f48be83670a06fc02b26d (table rebalance_test_table_2 [id=c723405d23a94457ac618ebd7a319e96]) successfully deleted
I20240501 07:58:17.057720 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.063819 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.066679 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.067148 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.078769 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.081399 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.082751 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.183115 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.185276 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.190093 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.191174 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.194993 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.195626 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.213508 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.430740 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.430912 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.431200 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.435647 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.438870 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.439958 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.447103 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.552268 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.554268 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.554283 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.558310 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.564234 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.581802 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.595913 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.682303 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.685302 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:17.685302 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.686805 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.698820 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.712757 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:17.715577 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d8d12c170da7423b82335f6f72d0a66d | 127.0.99.190:33335 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:36063 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3199fb0a9d4d448687cd049ae66601c9 | 127.0.99.132:35435 | HEALTHY | <none> | 2 | 0
827b6164a742460a8c2086c5cb1222f7 | 127.0.99.134:37621 | HEALTHY | <none> | 2 | 0
bf25fbb990fb46ccb496debdcc7cf91e | 127.0.99.129:45303 | HEALTHY | <none> | 3 | 0
f25b7c34fac14ab3bfe3c58386fcff04 | 127.0.99.133:35243 | HEALTHY | <none> | 2 | 0
f750f74c5d5b4295be7b3b3447f97797 | 127.0.99.135:45899 | HEALTHY | <none> | 2 | 0
fa963b55f7f04fdb9bd343e770f13dd5 | 127.0.99.130:38583 | HEALTHY | <none> | 3 | 0
fe0605ab798147669ec83b685fc7ace0 | 127.0.99.131:39669 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:45303
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:38583
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:39669
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35435
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:35243
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37621
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45899
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:45303
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:38583
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:39669
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35435
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:35243
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37621
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45899
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:36063 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:58:17.740876 398 log_verifier.cc:126] Checking tablet 30d99a0709f345848780a56d85a65186
I20240501 07:58:17.751899 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 30d99a0709f345848780a56d85a65186
I20240501 07:58:17.751983 398 log_verifier.cc:126] Checking tablet 56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:17.760185 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:17.760248 398 log_verifier.cc:126] Checking tablet 5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:17.769259 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:17.769325 398 log_verifier.cc:126] Checking tablet 6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:17.778746 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:17.778811 398 log_verifier.cc:126] Checking tablet 6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:17.788300 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:17.788401 398 log_verifier.cc:126] Checking tablet 7879c400867a4e4cbd2e51d323f78794
I20240501 07:58:17.796512 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7879c400867a4e4cbd2e51d323f78794
I20240501 07:58:17.796578 398 log_verifier.cc:126] Checking tablet 8b4663e0650c4209bbb88e87291af937
I20240501 07:58:17.805662 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8b4663e0650c4209bbb88e87291af937
I20240501 07:58:17.805727 398 log_verifier.cc:126] Checking tablet 914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:17.814786 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:17.814855 398 log_verifier.cc:126] Checking tablet 96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:17.823940 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:17.824000 398 log_verifier.cc:126] Checking tablet b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:17.832949 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:17.833011 398 log_verifier.cc:126] Checking tablet c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:17.842058 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:17.842125 398 log_verifier.cc:126] Checking tablet d6be2584c472449c8d570057f24d4ee4
I20240501 07:58:17.850872 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d6be2584c472449c8d570057f24d4ee4
I20240501 07:58:17.850935 398 log_verifier.cc:126] Checking tablet e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:17.860093 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:17.860172 398 log_verifier.cc:126] Checking tablet eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:17.869288 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:17.869365 398 log_verifier.cc:126] Checking tablet f920c1cf13e44a8fa8c167c5299c796f
I20240501 07:58:17.878540 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f920c1cf13e44a8fa8c167c5299c796f
I20240501 07:58:17.878616 398 log_verifier.cc:126] Checking tablet fe5420569ff445debb0e2ffdc7c4b1d1
I20240501 07:58:17.887677 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe5420569ff445debb0e2ffdc7c4b1d1
I20240501 07:58:17.889976 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--builtin_ntp_servers=127.0.99.148:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:58:18.001837 5949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:18.002102 5949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:18.002188 5949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:18.005589 5949 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:18.005681 5949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:18.005791 5949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:18.009644 5949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33335
--never_fsync=true
--heap_profile_path=/tmp/kudu.5949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:18.010586 5949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:18.011881 5949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:18.021337 5954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:18.022544 5949 server_base.cc:1034] running on GCE node
W20240501 07:58:18.023402 5955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:18.024127 5957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:18.024494 5949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:18.025071 5949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:18.026219 5949 hybrid_clock.cc:648] HybridClock initialized: now 1714550298026199 us; error 34 us; skew 500 ppm
I20240501 07:58:18.028043 5949 webserver.cc:460] Webserver started at http://127.0.99.136:42871/ using document root <none> and password file <none>
I20240501 07:58:18.028687 5949 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:18.028741 5949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:18.028916 5949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:18.030591 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "270f3488e88042659e077f18b385fd24"
format_stamp: "Formatted at 2024-05-01 07:58:18 on dist-test-slave-qm51"
I20240501 07:58:18.031121 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "270f3488e88042659e077f18b385fd24"
format_stamp: "Formatted at 2024-05-01 07:58:18 on dist-test-slave-qm51"
I20240501 07:58:18.034250 5949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:18.036381 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:18.037320 5949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:18.037429 5949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "270f3488e88042659e077f18b385fd24"
format_stamp: "Formatted at 2024-05-01 07:58:18 on dist-test-slave-qm51"
I20240501 07:58:18.037505 5949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:18.049618 5949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:18.050211 5949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:18.050361 5949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:18.050853 5949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:18.051723 5949 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:18.051779 5949 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:18.051821 5949 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:18.051851 5949 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:18.060410 5949 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:42033
I20240501 07:58:18.060462 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:42033 every 8 connection(s)
I20240501 07:58:18.061723 5949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:18.065733 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 5949
I20240501 07:58:18.065817 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:58:18.070814 6077 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33335
I20240501 07:58:18.071059 6077 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:18.071524 6077 heartbeater.cc:511] Master 127.0.99.190:33335 requested a full tablet report, sending...
I20240501 07:58:18.072440 2247 ts_manager.cc:194] Registered new tserver with Master: 270f3488e88042659e077f18b385fd24 (127.0.99.136:42033)
I20240501 07:58:18.072983 2247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:39299
I20240501 07:58:18.258329 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:18.258410 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.259476 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:18.259719 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.275949 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:18.276548 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.277635 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.278041 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.400730 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:18.400820 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.401736 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:18.406553 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:18.412071 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.413260 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.426539 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.438113 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:18.475584 2521 consensus_queue.cc:237] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
W20240501 07:58:18.480484 2454 consensus_peers.cc:485] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 -> Peer 270f3488e88042659e077f18b385fd24 (127.0.99.136:42033): Couldn't send request to peer 270f3488e88042659e077f18b385fd24. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e6eab741d4248c09234574cfe48fb2c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:19.074838 6077 heartbeater.cc:503] Master 127.0.99.190:33335 was elected leader, sending a full tablet report...
I20240501 07:58:19.086830 6183 ts_tablet_manager.cc:902] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Initiating tablet copy from peer fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:19.087558 6183 tablet_copy_client.cc:308] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:38583
I20240501 07:58:19.091437 2541 tablet_copy_service.cc:140] P fa963b55f7f04fdb9bd343e770f13dd5: Received BeginTabletCopySession request for tablet 5e6eab741d4248c09234574cfe48fb2c from peer 270f3488e88042659e077f18b385fd24 ({username='slave'} at 127.0.99.136:46141)
I20240501 07:58:19.091537 2541 tablet_copy_service.cc:161] P fa963b55f7f04fdb9bd343e770f13dd5: Beginning new tablet copy session on tablet 5e6eab741d4248c09234574cfe48fb2c from peer 270f3488e88042659e077f18b385fd24 at {username='slave'} at 127.0.99.136:46141: session id = 270f3488e88042659e077f18b385fd24-5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:19.092254 2541 tablet_copy_source_session.cc:205] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:19.095221 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6eab741d4248c09234574cfe48fb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:19.099697 6183 tablet_copy_client.cc:783] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:19.099890 6183 tablet_copy_client.cc:647] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:19.101415 6183 tablet_copy_client.cc:523] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:19.102942 6183 tablet_bootstrap.cc:492] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Bootstrap starting.
I20240501 07:58:19.107785 6183 log.cc:824] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:19.127149 6183 tablet_bootstrap.cc:492] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:19.127554 6183 tablet_bootstrap.cc:492] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Bootstrap complete.
I20240501 07:58:19.127849 6183 ts_tablet_manager.cc:1367] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240501 07:58:19.129843 6183 raft_consensus.cc:348] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } }
I20240501 07:58:19.130383 6183 raft_consensus.cc:729] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270f3488e88042659e077f18b385fd24, State: Initialized, Role: FOLLOWER
I20240501 07:58:19.130659 6183 consensus_queue.cc:260] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:19.131500 6183 ts_tablet_manager.cc:1398] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:19.132270 2541 tablet_copy_service.cc:342] P fa963b55f7f04fdb9bd343e770f13dd5: Request end of tablet copy session 270f3488e88042659e077f18b385fd24-5e6eab741d4248c09234574cfe48fb2c received from {username='slave'} at 127.0.99.136:46141
I20240501 07:58:19.132364 2541 tablet_copy_service.cc:433] P fa963b55f7f04fdb9bd343e770f13dd5: ending tablet copy session 270f3488e88042659e077f18b385fd24-5e6eab741d4248c09234574cfe48fb2c on tablet 5e6eab741d4248c09234574cfe48fb2c with peer 270f3488e88042659e077f18b385fd24
I20240501 07:58:19.563771 6031 raft_consensus.cc:1203] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:19.564980 6192 raft_consensus.cc:2932] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:19.565554 6031 raft_consensus.cc:2932] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:19.566381 6192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57056 (request call id 0) took 1091 ms. Trace:
I20240501 07:58:19.566478 6192 rpcz_store.cc:276] 0501 07:58:18.475182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 07:58:18.475277 (+ 95us) service_pool.cc:225] Handling call
0501 07:58:18.476039 (+ 762us) log.cc:842] Serialized 257 byte log entry
0501 07:58:19.566364 (+1090325us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":46,"raft.queue_time_us":147,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":158,"wal-append.run_cpu_time_us":318,"wal-append.run_wall_time_us":1000391}
I20240501 07:58:19.567724 2246 catalog_manager.cc:5484] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 reported cstate change: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:19.587754 2389 consensus_queue.cc:237] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
W20240501 07:58:19.592231 2323 consensus_peers.cc:485] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e -> Peer 270f3488e88042659e077f18b385fd24 (127.0.99.136:42033): Couldn't send request to peer 270f3488e88042659e077f18b385fd24. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56c2c5346ceb43a0b74ef2689e2d3d6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:19.978283 6220 ts_tablet_manager.cc:902] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Initiating tablet copy from peer bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:19.978775 6220 tablet_copy_client.cc:308] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45303
I20240501 07:58:19.982318 2410 tablet_copy_service.cc:140] P bf25fbb990fb46ccb496debdcc7cf91e: Received BeginTabletCopySession request for tablet 56c2c5346ceb43a0b74ef2689e2d3d6b from peer 270f3488e88042659e077f18b385fd24 ({username='slave'} at 127.0.99.136:59389)
I20240501 07:58:19.982462 2410 tablet_copy_service.cc:161] P bf25fbb990fb46ccb496debdcc7cf91e: Beginning new tablet copy session on tablet 56c2c5346ceb43a0b74ef2689e2d3d6b from peer 270f3488e88042659e077f18b385fd24 at {username='slave'} at 127.0.99.136:59389: session id = 270f3488e88042659e077f18b385fd24-56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:19.983311 2410 tablet_copy_source_session.cc:205] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:19.984213 6220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c2c5346ceb43a0b74ef2689e2d3d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:19.985958 6220 tablet_copy_client.cc:783] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:19.986109 6220 tablet_copy_client.cc:647] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:19.987154 6220 tablet_copy_client.cc:523] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:19.988094 6220 tablet_bootstrap.cc:492] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Bootstrap starting.
I20240501 07:58:20.004546 6220 tablet_bootstrap.cc:492] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:58:20.004817 6220 tablet_bootstrap.cc:492] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Bootstrap complete.
I20240501 07:58:20.004974 6220 ts_tablet_manager.cc:1367] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240501 07:58:20.005470 6220 raft_consensus.cc:348] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } }
I20240501 07:58:20.005683 6220 raft_consensus.cc:729] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270f3488e88042659e077f18b385fd24, State: Initialized, Role: FOLLOWER
I20240501 07:58:20.005779 6220 consensus_queue.cc:260] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.006006 6220 ts_tablet_manager.cc:1398] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:20.006599 2410 tablet_copy_service.cc:342] P bf25fbb990fb46ccb496debdcc7cf91e: Request end of tablet copy session 270f3488e88042659e077f18b385fd24-56c2c5346ceb43a0b74ef2689e2d3d6b received from {username='slave'} at 127.0.99.136:59389
I20240501 07:58:20.006701 2410 tablet_copy_service.cc:433] P bf25fbb990fb46ccb496debdcc7cf91e: ending tablet copy session 270f3488e88042659e077f18b385fd24-56c2c5346ceb43a0b74ef2689e2d3d6b on tablet 56c2c5346ceb43a0b74ef2689e2d3d6b with peer 270f3488e88042659e077f18b385fd24
I20240501 07:58:20.475883 6031 raft_consensus.cc:1203] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:20.476883 6227 raft_consensus.cc:2932] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:20.477289 6031 raft_consensus.cc:2932] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:20.479647 2247 catalog_manager.cc:5484] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e reported cstate change: config changed from index -1 to 4, VOTER 270f3488e88042659e077f18b385fd24 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:20.535929 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.536971 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:20.538507 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.541455 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.550418 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.559844 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.568894 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.583698 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.605057 2389 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56c2c5346ceb43a0b74ef2689e2d3d6b"
dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40130
I20240501 07:58:20.605206 2389 raft_consensus.cc:595] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:20.654596 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:20.655301 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.658267 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.659847 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.670946 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.671217 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.677843 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.683830 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.765134 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.769547 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.770680 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.772037 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.778529 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.778563 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.787236 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:20.796905 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.825469 2521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e6eab741d4248c09234574cfe48fb2c"
dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57090
I20240501 07:58:20.825618 2521 raft_consensus.cc:595] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:20.874198 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.878736 2369 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.878727 6011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:20.879139 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.887575 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.889674 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.909324 2501 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:20.909709 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:20.925920 6227 raft_consensus.cc:979] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: : Instructing follower 270f3488e88042659e077f18b385fd24 to start an election
I20240501 07:58:20.926029 6227 raft_consensus.cc:1067] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Signalling peer 270f3488e88042659e077f18b385fd24 to start an election
I20240501 07:58:20.926626 6031 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56c2c5346ceb43a0b74ef2689e2d3d6b"
dest_uuid: "270f3488e88042659e077f18b385fd24"
from {username='slave'} at 127.0.99.129:38801
I20240501 07:58:20.926851 6031 raft_consensus.cc:482] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:20.926962 6031 raft_consensus.cc:3037] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:20.928158 6031 raft_consensus.cc:504] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.928962 6031 leader_election.cc:288] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [CANDIDATE]: Term 2 election: Requested vote from peers bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303)
I20240501 07:58:20.929291 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c2c5346ceb43a0b74ef2689e2d3d6b" candidate_uuid: "270f3488e88042659e077f18b385fd24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bf25fbb990fb46ccb496debdcc7cf91e"
I20240501 07:58:20.929404 2389 raft_consensus.cc:3032] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:20.929432 2389 raft_consensus.cc:729] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf25fbb990fb46ccb496debdcc7cf91e, State: Running, Role: LEADER
I20240501 07:58:20.929554 2389 consensus_queue.cc:260] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.929678 2389 raft_consensus.cc:3037] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:20.930444 2389 raft_consensus.cc:2454] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 270f3488e88042659e077f18b385fd24 in term 2.
I20240501 07:58:20.930830 5966 leader_election.cc:302] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 270f3488e88042659e077f18b385fd24, bf25fbb990fb46ccb496debdcc7cf91e; no voters:
I20240501 07:58:20.931226 6191 raft_consensus.cc:2781] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:20.931409 6191 raft_consensus.cc:686] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 2 LEADER]: Becoming Leader. State: Replica: 270f3488e88042659e077f18b385fd24, State: Running, Role: LEADER
I20240501 07:58:20.931691 6191 consensus_queue.cc:237] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.933867 2247 catalog_manager.cc:5484] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 reported cstate change: term changed from 1 to 2, leader changed from bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) to 270f3488e88042659e077f18b385fd24 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "270f3488e88042659e077f18b385fd24" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:20.956436 6395 raft_consensus.cc:979] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: : Instructing follower 270f3488e88042659e077f18b385fd24 to start an election
I20240501 07:58:20.956534 6395 raft_consensus.cc:1067] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Signalling peer 270f3488e88042659e077f18b385fd24 to start an election
I20240501 07:58:20.956959 6031 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e6eab741d4248c09234574cfe48fb2c"
dest_uuid: "270f3488e88042659e077f18b385fd24"
from {username='slave'} at 127.0.99.130:37977
I20240501 07:58:20.957082 6031 raft_consensus.cc:482] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:20.957130 6031 raft_consensus.cc:3037] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:20.957914 6031 raft_consensus.cc:504] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.958231 6031 leader_election.cc:288] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [CANDIDATE]: Term 2 election: Requested vote from peers fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583)
I20240501 07:58:20.958592 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6eab741d4248c09234574cfe48fb2c" candidate_uuid: "270f3488e88042659e077f18b385fd24" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa963b55f7f04fdb9bd343e770f13dd5"
I20240501 07:58:20.958707 2521 raft_consensus.cc:3032] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:20.958758 2521 raft_consensus.cc:729] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa963b55f7f04fdb9bd343e770f13dd5, State: Running, Role: LEADER
I20240501 07:58:20.958938 2521 consensus_queue.cc:260] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.959097 2521 raft_consensus.cc:3037] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:20.959813 2521 raft_consensus.cc:2454] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 270f3488e88042659e077f18b385fd24 in term 2.
I20240501 07:58:20.960095 5966 leader_election.cc:302] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 270f3488e88042659e077f18b385fd24, fa963b55f7f04fdb9bd343e770f13dd5; no voters:
I20240501 07:58:20.960263 6191 raft_consensus.cc:2781] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:20.960361 6191 raft_consensus.cc:686] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 2 LEADER]: Becoming Leader. State: Replica: 270f3488e88042659e077f18b385fd24, State: Running, Role: LEADER
I20240501 07:58:20.960484 6191 consensus_queue.cc:237] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:20.961849 2247 catalog_manager.cc:5484] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 reported cstate change: term changed from 1 to 2, leader changed from fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) to 270f3488e88042659e077f18b385fd24 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "270f3488e88042659e077f18b385fd24" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 } } peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:21.190399 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.191699 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.195524 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.195729 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.196467 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.208941 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.210306 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.211117 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.297611 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.301698 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.304585 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.306763 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.307569 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.307917 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.321992 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.341523 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.394914 2389 raft_consensus.cc:1261] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Refusing update from remote peer 270f3488e88042659e077f18b385fd24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:21.395507 6191 consensus_queue.cc:1035] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf25fbb990fb46ccb496debdcc7cf91e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:21.448318 2521 raft_consensus.cc:1261] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Refusing update from remote peer 270f3488e88042659e077f18b385fd24: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:21.448825 6394 consensus_queue.cc:1035] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa963b55f7f04fdb9bd343e770f13dd5" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:21.619287 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.619311 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.621665 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.625443 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.626226 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.632282 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.637370 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.644485 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.689671 6031 consensus_queue.cc:237] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:21.690322 6191 raft_consensus.cc:2932] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:21.692668 2240 catalog_manager.cc:5484] T 56c2c5346ceb43a0b74ef2689e2d3d6b P 270f3488e88042659e077f18b385fd24 reported cstate change: config changed from index 4 to 6, VOTER bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "270f3488e88042659e077f18b385fd24" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:21.696502 2369 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c2c5346ceb43a0b74ef2689e2d3d6b with delete_type TABLET_DATA_TOMBSTONED (TS bf25fbb990fb46ccb496debdcc7cf91e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45782
I20240501 07:58:21.696836 6539 tablet_replica.cc:330] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: stopping tablet replica
I20240501 07:58:21.697016 6539 raft_consensus.cc:2229] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:21.697153 6539 raft_consensus.cc:2258] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:21.698788 6539 ts_tablet_manager.cc:1875] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:21.701387 6539 ts_tablet_manager.cc:1888] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:21.701467 6539 log.cc:1196] T 56c2c5346ceb43a0b74ef2689e2d3d6b P bf25fbb990fb46ccb496debdcc7cf91e: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:21.702087 2234 catalog_manager.cc:4830] TS bf25fbb990fb46ccb496debdcc7cf91e (127.0.99.129:45303): tablet 56c2c5346ceb43a0b74ef2689e2d3d6b (table rebalance_test_table_3 [id=a3667a8bcce94c05ba37d2de9899485d]) successfully deleted
I20240501 07:58:21.752285 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.753314 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.753787 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.763221 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.770571 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.771240 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.771502 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.799614 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:21.833083 6031 consensus_queue.cc:237] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } }
I20240501 07:58:21.833704 6191 raft_consensus.cc:2932] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:21.835940 2247 catalog_manager.cc:5484] T 5e6eab741d4248c09234574cfe48fb2c P 270f3488e88042659e077f18b385fd24 reported cstate change: config changed from index 4 to 6, VOTER fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "270f3488e88042659e077f18b385fd24" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "270f3488e88042659e077f18b385fd24" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42033 } } }
I20240501 07:58:21.838241 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6eab741d4248c09234574cfe48fb2c with delete_type TABLET_DATA_TOMBSTONED (TS fa963b55f7f04fdb9bd343e770f13dd5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33458
I20240501 07:58:21.838570 6588 tablet_replica.cc:330] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: stopping tablet replica
I20240501 07:58:21.838685 6588 raft_consensus.cc:2229] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:21.838771 6588 raft_consensus.cc:2258] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:21.840162 6588 ts_tablet_manager.cc:1875] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:21.842032 6588 ts_tablet_manager.cc:1888] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:21.842118 6588 log.cc:1196] T 5e6eab741d4248c09234574cfe48fb2c P fa963b55f7f04fdb9bd343e770f13dd5: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:21.842608 2234 catalog_manager.cc:4830] TS fa963b55f7f04fdb9bd343e770f13dd5 (127.0.99.130:38583): tablet 5e6eab741d4248c09234574cfe48fb2c (table rebalance_test_table_1 [id=dc8bf09ec7d4480bb498725bb829e50d]) successfully deleted
I20240501 07:58:22.114117 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.114321 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.114629 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.115370 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.116624 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.128077 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.128537 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.149283 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.228890 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.229194 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.229430 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.234846 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.247359 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.249316 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.261453 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.261804 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.346648 3906 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.348783 3634 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.350056 4037 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.351253 6011 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.354344 3758 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.361577 2501 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.363287 3503 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:22.387683 2369 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d8d12c170da7423b82335f6f72d0a66d | 127.0.99.190:33335 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:36063 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
270f3488e88042659e077f18b385fd24 | 127.0.99.136:42033 | HEALTHY | <none> | 2 | 0
3199fb0a9d4d448687cd049ae66601c9 | 127.0.99.132:35435 | HEALTHY | <none> | 2 | 0
827b6164a742460a8c2086c5cb1222f7 | 127.0.99.134:37621 | HEALTHY | <none> | 2 | 0
bf25fbb990fb46ccb496debdcc7cf91e | 127.0.99.129:45303 | HEALTHY | <none> | 2 | 0
f25b7c34fac14ab3bfe3c58386fcff04 | 127.0.99.133:35243 | HEALTHY | <none> | 2 | 0
f750f74c5d5b4295be7b3b3447f97797 | 127.0.99.135:45899 | HEALTHY | <none> | 2 | 0
fa963b55f7f04fdb9bd343e770f13dd5 | 127.0.99.130:38583 | HEALTHY | <none> | 2 | 0
fe0605ab798147669ec83b685fc7ace0 | 127.0.99.131:39669 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:45303
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:38583
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:39669
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:35435
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:35243
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37621
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45899
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:42033
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:45303
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:38583
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:39669
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:35435
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:35243
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37621
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45899
server_dump_info_path | /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:42033
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:36063 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:58:22.409677 398 log_verifier.cc:126] Checking tablet 30d99a0709f345848780a56d85a65186
I20240501 07:58:22.415931 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 30d99a0709f345848780a56d85a65186
I20240501 07:58:22.415995 398 log_verifier.cc:126] Checking tablet 56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:22.425648 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 56c2c5346ceb43a0b74ef2689e2d3d6b
I20240501 07:58:22.425724 398 log_verifier.cc:126] Checking tablet 5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:22.434664 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5e6eab741d4248c09234574cfe48fb2c
I20240501 07:58:22.434736 398 log_verifier.cc:126] Checking tablet 6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:22.440438 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6a0be484474f4cde809c8757a87c1bea
I20240501 07:58:22.440496 398 log_verifier.cc:126] Checking tablet 6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:22.446187 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6eb7d8d7017f48be83670a06fc02b26d
I20240501 07:58:22.446244 398 log_verifier.cc:126] Checking tablet 7879c400867a4e4cbd2e51d323f78794
I20240501 07:58:22.450876 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7879c400867a4e4cbd2e51d323f78794
I20240501 07:58:22.450935 398 log_verifier.cc:126] Checking tablet 8b4663e0650c4209bbb88e87291af937
I20240501 07:58:22.456689 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8b4663e0650c4209bbb88e87291af937
I20240501 07:58:22.456748 398 log_verifier.cc:126] Checking tablet 914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:22.462436 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 914df5ce34d5464a9a3876fb0cc472f1
I20240501 07:58:22.462491 398 log_verifier.cc:126] Checking tablet 96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:22.468415 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 96af3865d3ef469ea5b24cba20519bb3
I20240501 07:58:22.468470 398 log_verifier.cc:126] Checking tablet b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:22.474166 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b6c35e3ea72543f1bd874e72f708cdf2
I20240501 07:58:22.474225 398 log_verifier.cc:126] Checking tablet c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:22.479924 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c7c7acd9cd91461197fc2e2c86bd570f
I20240501 07:58:22.479983 398 log_verifier.cc:126] Checking tablet d6be2584c472449c8d570057f24d4ee4
I20240501 07:58:22.485760 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d6be2584c472449c8d570057f24d4ee4
I20240501 07:58:22.485824 398 log_verifier.cc:126] Checking tablet e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:22.491585 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e734066a97d145338ab9c525d5d4c1dc
I20240501 07:58:22.491641 398 log_verifier.cc:126] Checking tablet eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:22.497323 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet eff829f74ea44c7c9cefa72260ac2165
I20240501 07:58:22.497385 398 log_verifier.cc:126] Checking tablet f920c1cf13e44a8fa8c167c5299c796f
I20240501 07:58:22.502753 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f920c1cf13e44a8fa8c167c5299c796f
I20240501 07:58:22.502804 398 log_verifier.cc:126] Checking tablet fe5420569ff445debb0e2ffdc7c4b1d1
I20240501 07:58:22.508234 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe5420569ff445debb0e2ffdc7c4b1d1
I20240501 07:58:22.508700 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2308
I20240501 07:58:22.515307 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2439
I20240501 07:58:22.521724 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3441
I20240501 07:58:22.527302 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3572
I20240501 07:58:22.533352 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3703
I20240501 07:58:22.538882 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3836
I20240501 07:58:22.544788 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 3975
I20240501 07:58:22.552352 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 5949
I20240501 07:58:22.558022 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 2217
2024-05-01T07:58:22Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (27214 ms)
[----------] 1 test from RebalanceParamTest (27214 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 07:58:22.598443 398 test_util.cc:268] Using random seed: -641964766
I20240501 07:58:22.599537 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:22.599589 398 ts_itest-base.cc:113] --------------
I20240501 07:58:22.599637 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 07:58:22.599655 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:22.599679 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:22Z Disabled control of system clock
I20240501 07:58:22.607878 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42471
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42471
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 07:58:22.712764 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:22.712989 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:22.713037 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:22.716135 6706 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:22.716208 6706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:22.716233 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:22.716250 6706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:22.716284 6706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:22.719928 6706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42471
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:22.720871 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:22.721982 6706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:22.729755 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:22.730485 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:22.731292 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:22.732529 6706 server_base.cc:1034] running on GCE node
I20240501 07:58:22.733083 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:22.734040 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:22.735195 6706 hybrid_clock.cc:648] HybridClock initialized: now 1714550302735184 us; error 29 us; skew 500 ppm
I20240501 07:58:22.737149 6706 webserver.cc:460] Webserver started at http://127.0.99.190:33303/ using document root <none> and password file <none>
I20240501 07:58:22.737696 6706 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:22.737757 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:22.737927 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:22.739620 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d467cffe72b54dd1ac420321915f63b7"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.740097 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d467cffe72b54dd1ac420321915f63b7"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.743391 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:22.745597 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:22.746810 6706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:22.746935 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d467cffe72b54dd1ac420321915f63b7"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.747071 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:22.770083 6706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:22.770749 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:22.770902 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:22.777220 6706 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42471
I20240501 07:58:22.777237 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42471 every 8 connection(s)
I20240501 07:58:22.778239 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:22.780894 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:22.782987 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 6706
I20240501 07:58:22.783162 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:22.786080 6773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Bootstrap starting.
I20240501 07:58:22.788044 6773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:22.788723 6773 log.cc:824] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:22.790309 6773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: No bootstrap required, opened a new log
I20240501 07:58:22.792824 6773 raft_consensus.cc:348] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } }
I20240501 07:58:22.792985 6773 raft_consensus.cc:374] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:22.793018 6773 raft_consensus.cc:729] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d467cffe72b54dd1ac420321915f63b7, State: Initialized, Role: FOLLOWER
I20240501 07:58:22.793442 6773 consensus_queue.cc:260] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } }
I20240501 07:58:22.793588 6773 raft_consensus.cc:388] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:22.793650 6773 raft_consensus.cc:482] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:22.793745 6773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:22.794407 6773 raft_consensus.cc:504] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } }
I20240501 07:58:22.794692 6773 leader_election.cc:302] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d467cffe72b54dd1ac420321915f63b7; no voters:
I20240501 07:58:22.794924 6773 leader_election.cc:288] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:22.795348 6778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:22.795513 6778 raft_consensus.cc:686] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [term 1 LEADER]: Becoming Leader. State: Replica: d467cffe72b54dd1ac420321915f63b7, State: Running, Role: LEADER
I20240501 07:58:22.795804 6778 consensus_queue.cc:237] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } }
I20240501 07:58:22.796005 6773 sys_catalog.cc:564] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:22.797187 6780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d467cffe72b54dd1ac420321915f63b7. Latest consensus state: current_term: 1 leader_uuid: "d467cffe72b54dd1ac420321915f63b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } } }
I20240501 07:58:22.797327 6780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:22.797219 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d467cffe72b54dd1ac420321915f63b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d467cffe72b54dd1ac420321915f63b7" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42471 } } }
I20240501 07:58:22.797416 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7 [sys.catalog]: This master's current role is: LEADER
W20240501 07:58:22.800352 6793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:22.800422 6793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:22.800532 6787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:22.801404 6787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:22.806262 6787 catalog_manager.cc:1347] Generated new cluster ID: a027f3ea9a94490f9dce3b9be778ddc2
I20240501 07:58:22.806320 6787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:22.812068 6787 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:22.812938 6787 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:22.817849 6787 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Generated new TSK 0
I20240501 07:58:22.818444 6787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:22.825191 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:58:22.929939 6797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:22.930155 6797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:22.930198 6797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:22.933216 6797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:22.933279 6797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:22.933357 6797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:22.936882 6797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.6797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:22.937739 6797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:22.938812 6797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:22.947760 6805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:22.948920 6802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:22.950047 6803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:22.951092 6797 server_base.cc:1034] running on GCE node
I20240501 07:58:22.951503 6797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:22.952167 6797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:22.953332 6797 hybrid_clock.cc:648] HybridClock initialized: now 1714550302953310 us; error 42 us; skew 500 ppm
I20240501 07:58:22.955380 6797 webserver.cc:460] Webserver started at http://127.0.99.129:44043/ using document root <none> and password file <none>
I20240501 07:58:22.955960 6797 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:22.956020 6797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:22.956198 6797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:22.957880 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1a9029bee6a144938f68b363e4cca162"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.958345 6797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1a9029bee6a144938f68b363e4cca162"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.961725 6797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:58:22.963918 6811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:22.964874 6797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:22.964984 6797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1a9029bee6a144938f68b363e4cca162"
format_stamp: "Formatted at 2024-05-01 07:58:22 on dist-test-slave-qm51"
I20240501 07:58:22.965065 6797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.002753 6797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.003499 6797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.003741 6797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.004344 6797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.005210 6797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.005259 6797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.005318 6797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.005353 6797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.013639 6797 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37209
I20240501 07:58:23.013676 6924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37209 every 8 connection(s)
I20240501 07:58:23.014561 6797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:23.021211 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 6797
I20240501 07:58:23.021342 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:23.022956 6925 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.023209 6925 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.023646 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.023716 6925 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.025172 6737 ts_manager.cc:194] Registered new tserver with Master: 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:23.026885 6737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48791
W20240501 07:58:23.130136 6928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:23.130347 6928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:23.130389 6928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:23.133442 6928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:23.133510 6928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:23.133599 6928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:23.137115 6928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.6928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:23.138032 6928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:23.139163 6928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:23.147958 6933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.148912 6936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.149883 6934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:23.151079 6928 server_base.cc:1034] running on GCE node
I20240501 07:58:23.151526 6928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:23.152118 6928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:23.153304 6928 hybrid_clock.cc:648] HybridClock initialized: now 1714550303153276 us; error 41 us; skew 500 ppm
I20240501 07:58:23.155356 6928 webserver.cc:460] Webserver started at http://127.0.99.130:40009/ using document root <none> and password file <none>
I20240501 07:58:23.155876 6928 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:23.155929 6928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:23.156085 6928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:23.157676 6928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9a3888360d4d4ace96b6b909c160bfc2"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.158090 6928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9a3888360d4d4ace96b6b909c160bfc2"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.161329 6928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 07:58:23.163479 6942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.164491 6928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:23.164587 6928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9a3888360d4d4ace96b6b909c160bfc2"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.164695 6928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.176312 6928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.176868 6928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.177021 6928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.177548 6928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.178380 6928 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.178432 6928 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.178471 6928 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.178488 6928 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.186829 6928 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39675
I20240501 07:58:23.186858 7055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39675 every 8 connection(s)
I20240501 07:58:23.187716 6928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:23.189837 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 6928
I20240501 07:58:23.189944 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:23.192103 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.197280 7056 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.197515 7056 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.197937 7056 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.198916 6737 ts_manager.cc:194] Registered new tserver with Master: 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:23.199810 6737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:33619
W20240501 07:58:23.297343 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:23.297542 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:23.297585 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:23.300650 7059 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:23.300714 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:23.300779 7059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:23.304220 7059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:23.305011 7059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:23.306039 7059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:23.315135 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.315886 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.316762 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:23.317633 7059 server_base.cc:1034] running on GCE node
I20240501 07:58:23.318063 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:23.318604 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:23.319746 7059 hybrid_clock.cc:648] HybridClock initialized: now 1714550303319728 us; error 35 us; skew 500 ppm
I20240501 07:58:23.321821 7059 webserver.cc:460] Webserver started at http://127.0.99.131:35119/ using document root <none> and password file <none>
I20240501 07:58:23.322319 7059 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:23.322376 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:23.322520 7059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:23.324147 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fc946354b5d04503a56122e59fc320be"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.324585 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fc946354b5d04503a56122e59fc320be"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.327798 7059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:23.329779 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.330736 7059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:23.330829 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fc946354b5d04503a56122e59fc320be"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.330899 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.352965 7059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.353513 7059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.353642 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.354103 7059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.354919 7059 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.354964 7059 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.355031 7059 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.355067 7059 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.364033 7059 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:37107
I20240501 07:58:23.364059 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:37107 every 8 connection(s)
I20240501 07:58:23.364979 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:23.368113 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7059
I20240501 07:58:23.368219 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:23.370453 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.374557 7187 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.374763 7187 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.375224 7187 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.376149 6737 ts_manager.cc:194] Registered new tserver with Master: fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:23.376931 6737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:53941
W20240501 07:58:23.477548 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:23.477761 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:23.477806 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:23.480737 7190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:23.480798 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:23.480860 7190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:23.484068 7190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:23.484833 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:23.485957 7190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:23.494848 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.495620 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.496415 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:23.497591 7190 server_base.cc:1034] running on GCE node
I20240501 07:58:23.498000 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:23.498585 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:23.499735 7190 hybrid_clock.cc:648] HybridClock initialized: now 1714550303499722 us; error 28 us; skew 500 ppm
I20240501 07:58:23.505419 7190 webserver.cc:460] Webserver started at http://127.0.99.132:33831/ using document root <none> and password file <none>
I20240501 07:58:23.505978 7190 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:23.506039 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:23.506217 7190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:23.507918 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "15ab96f6477b4291932409b80af07213"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.508388 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "15ab96f6477b4291932409b80af07213"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.511688 7190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:23.513764 7204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.514704 7190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:23.514806 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "15ab96f6477b4291932409b80af07213"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.514881 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.526904 7190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.527531 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.527700 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.528175 7190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.529031 7190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.529085 7190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.529147 7190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.529177 7190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.538210 7190 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41309
I20240501 07:58:23.538244 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41309 every 8 connection(s)
I20240501 07:58:23.539216 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:23.546500 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7190
I20240501 07:58:23.546617 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:23.548138 7318 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.548424 7318 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.548864 7318 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.549007 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.549850 6736 ts_manager.cc:194] Registered new tserver with Master: 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:23.550417 6736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50707
W20240501 07:58:23.654889 7321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:23.655138 7321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:23.655181 7321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:23.658159 7321 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:23.658226 7321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:23.658310 7321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:23.661895 7321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:23.662777 7321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:23.663956 7321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:23.672998 7327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.674156 7326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.674950 7329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:23.675784 7321 server_base.cc:1034] running on GCE node
I20240501 07:58:23.676214 7321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:23.676776 7321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:23.677947 7321 hybrid_clock.cc:648] HybridClock initialized: now 1714550303677940 us; error 30 us; skew 500 ppm
I20240501 07:58:23.679965 7321 webserver.cc:460] Webserver started at http://127.0.99.133:35241/ using document root <none> and password file <none>
I20240501 07:58:23.680557 7321 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:23.680624 7321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:23.680814 7321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:23.682543 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "49adb9e99822494196ed22044bfed81f"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.683070 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "49adb9e99822494196ed22044bfed81f"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.686517 7321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:23.688831 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.689790 7321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:23.689898 7321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "49adb9e99822494196ed22044bfed81f"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.689986 7321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.709666 7321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.710315 7321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.710500 7321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.711104 7321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.712061 7321 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.712116 7321 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.712183 7321 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.712213 7321 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.721467 7321 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46797
I20240501 07:58:23.721498 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46797 every 8 connection(s)
I20240501 07:58:23.722476 7321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:23.725102 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7321
I20240501 07:58:23.725234 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:23.727573 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.731778 7449 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.731990 7449 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.732447 7449 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.733399 6736 ts_manager.cc:194] Registered new tserver with Master: 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:23.733942 6736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57979
W20240501 07:58:23.837023 7452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:23.837261 7452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:23.837334 7452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:23.840399 7452 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:23.840483 7452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:23.840600 7452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:23.844223 7452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:23.845165 7452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:23.846290 7452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:23.855515 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.856233 7460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:23.857439 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:23.858587 7452 server_base.cc:1034] running on GCE node
I20240501 07:58:23.859052 7452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:23.859648 7452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:23.860803 7452 hybrid_clock.cc:648] HybridClock initialized: now 1714550303860786 us; error 33 us; skew 500 ppm
I20240501 07:58:23.862802 7452 webserver.cc:460] Webserver started at http://127.0.99.134:39949/ using document root <none> and password file <none>
I20240501 07:58:23.863411 7452 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:23.863473 7452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:23.863644 7452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:23.865296 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fa6964452854415a85ad72f8639768d6"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.865769 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fa6964452854415a85ad72f8639768d6"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.869290 7452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:23.871519 7466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.872539 7452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:23.872649 7452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa6964452854415a85ad72f8639768d6"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:23.872740 7452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:23.889788 7452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:23.890426 7452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:23.890585 7452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:23.891139 7452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:23.892022 7452 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:23.892072 7452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.892134 7452 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:23.892160 7452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:23.900671 7452 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42153
I20240501 07:58:23.900722 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42153 every 8 connection(s)
I20240501 07:58:23.901584 7452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:23.903582 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7452
I20240501 07:58:23.903683 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:58:23.905776 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:23.910841 7580 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:23.911100 7580 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:23.911556 7580 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:23.912554 6736 ts_manager.cc:194] Registered new tserver with Master: fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:23.913399 6736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:52061
W20240501 07:58:24.012971 7583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:24.013211 7583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:24.013255 7583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:24.016347 7583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:24.016417 7583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:24.016495 7583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:24.020121 7583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:24.020988 7583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:24.022076 7583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:24.029248 6925 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
W20240501 07:58:24.030844 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:24.031949 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:24.032984 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:24.033906 7583 server_base.cc:1034] running on GCE node
I20240501 07:58:24.034387 7583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:24.034967 7583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:24.036170 7583 hybrid_clock.cc:648] HybridClock initialized: now 1714550304036154 us; error 39 us; skew 500 ppm
I20240501 07:58:24.038255 7583 webserver.cc:460] Webserver started at http://127.0.99.135:32771/ using document root <none> and password file <none>
I20240501 07:58:24.038811 7583 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:24.038885 7583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:24.039106 7583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:24.040820 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ac6c8bc1b74b483791e262abee7862cf"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.041291 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ac6c8bc1b74b483791e262abee7862cf"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.044569 7583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:24.046787 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.047896 7583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:24.048018 7583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ac6c8bc1b74b483791e262abee7862cf"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.048105 7583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:24.083675 7583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:24.084288 7583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:24.084451 7583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:24.084977 7583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:24.085835 7583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:24.085882 7583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.085942 7583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:24.085973 7583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.094336 7583 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:36013
I20240501 07:58:24.094383 7710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:36013 every 8 connection(s)
I20240501 07:58:24.095329 7583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:24.101832 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7583
I20240501 07:58:24.101969 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:58:24.104146 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:24.104187 7711 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:24.104405 7711 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:24.104939 7711 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:24.105994 6736 ts_manager.cc:194] Registered new tserver with Master: ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:24.106628 6736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:52629
I20240501 07:58:24.201658 7056 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
W20240501 07:58:24.214637 7714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:24.214855 7714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:24.214931 7714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:24.217921 7714 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:24.217990 7714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:24.218106 7714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:24.221941 7714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.7714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:24.222810 7714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:24.223973 7714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:24.232933 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:24.234019 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:24.234834 7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:24.235787 7714 server_base.cc:1034] running on GCE node
I20240501 07:58:24.236177 7714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:24.236713 7714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:24.237891 7714 hybrid_clock.cc:648] HybridClock initialized: now 1714550304237882 us; error 28 us; skew 500 ppm
I20240501 07:58:24.239984 7714 webserver.cc:460] Webserver started at http://127.0.99.136:41661/ using document root <none> and password file <none>
I20240501 07:58:24.240542 7714 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:24.240607 7714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:24.240793 7714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:24.242476 7714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7fa3a88c25964445975bb47e0feed460"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.242976 7714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7fa3a88c25964445975bb47e0feed460"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.246471 7714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:24.248773 7728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.249848 7714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:24.249962 7714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7fa3a88c25964445975bb47e0feed460"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:24.250061 7714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:24.266784 7714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:24.267489 7714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:24.267652 7714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:24.268260 7714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:24.269167 7714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:24.269241 7714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.269315 7714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:24.269349 7714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:24.277595 7714 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35111
I20240501 07:58:24.277622 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35111 every 8 connection(s)
I20240501 07:58:24.278524 7714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:24.280319 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 7714
I20240501 07:58:24.280437 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:58:24.287331 7842 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:24.287544 7842 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:24.287977 7842 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:24.288895 6736 ts_manager.cc:194] Registered new tserver with Master: 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:24.289441 6736 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34631
I20240501 07:58:24.294721 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:24.304121 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:24.304193 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7321
I20240501 07:58:24.309784 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7452
I20240501 07:58:24.315061 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7583
I20240501 07:58:24.320286 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7714
I20240501 07:58:24.378808 7187 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
I20240501 07:58:24.552482 7318 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
I20240501 07:58:28.077090 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:28.097895 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 48b6f304fb9141fea296a48d35eb33ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.097827 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7033ccd32d43f49df11d96a8752276 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.098245 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc04acac844b5ab0995fad69e51bbe (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.098847 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b6f304fb9141fea296a48d35eb33ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.098242 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 48b6f304fb9141fea296a48d35eb33ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.098992 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc04acac844b5ab0995fad69e51bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.099326 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7033ccd32d43f49df11d96a8752276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.099324 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7033ccd32d43f49df11d96a8752276 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.099537 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7033ccd32d43f49df11d96a8752276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.103109 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 48b6f304fb9141fea296a48d35eb33ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.103719 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b6f304fb9141fea296a48d35eb33ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.105034 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7033ccd32d43f49df11d96a8752276 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.105265 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7033ccd32d43f49df11d96a8752276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.105320 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b6f304fb9141fea296a48d35eb33ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.105784 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc04acac844b5ab0995fad69e51bbe (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.106062 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc04acac844b5ab0995fad69e51bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.106228 7863 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:28.108420 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 49bc04acac844b5ab0995fad69e51bbe (DEFAULT_TABLE table=rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.108561 7863 tablet_bootstrap.cc:654] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.108697 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bc04acac844b5ab0995fad69e51bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.109365 7863 log.cc:824] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.110945 7864 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:28.112303 7862 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:28.113027 7866 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:28.113221 7864 tablet_bootstrap.cc:654] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.114008 7864 log.cc:824] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.114074 7863 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:28.114212 7862 tablet_bootstrap.cc:654] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.114414 7863 ts_tablet_manager.cc:1367] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 07:58:28.114841 7866 tablet_bootstrap.cc:654] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.114945 7862 log.cc:824] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.115561 7866 log.cc:824] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.115648 7864 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:28.115813 7864 ts_tablet_manager.cc:1367] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 07:58:28.116605 7862 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:28.116770 7862 ts_tablet_manager.cc:1367] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 07:58:28.117007 7866 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:28.117158 7866 ts_tablet_manager.cc:1367] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 07:58:28.119598 7863 raft_consensus.cc:348] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.119967 7863 raft_consensus.cc:374] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.120085 7863 raft_consensus.cc:729] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.120332 7864 raft_consensus.cc:348] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.120532 7864 raft_consensus.cc:374] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.120585 7864 raft_consensus.cc:729] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.120606 7862 raft_consensus.cc:348] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.120792 7862 raft_consensus.cc:374] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.120846 7862 raft_consensus.cc:729] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.120934 7863 consensus_queue.cc:260] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.120980 7864 consensus_queue.cc:260] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.121116 7866 raft_consensus.cc:348] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.121307 7866 raft_consensus.cc:374] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.121363 7866 raft_consensus.cc:729] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.121274 7862 consensus_queue.cc:260] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.121739 7866 consensus_queue.cc:260] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.121879 7864 ts_tablet_manager.cc:1398] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 07:58:28.122020 7862 ts_tablet_manager.cc:1398] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 07:58:28.122130 7863 ts_tablet_manager.cc:1398] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 07:58:28.122251 7864 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:28.122467 7866 ts_tablet_manager.cc:1398] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20240501 07:58:28.122664 7863 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:28.123932 7866 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:28.124168 7864 tablet_bootstrap.cc:654] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.124191 7862 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:28.125969 7864 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:28.126060 7862 tablet_bootstrap.cc:654] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.126065 7864 ts_tablet_manager.cc:1367] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:28.126101 7866 tablet_bootstrap.cc:654] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.126673 7864 raft_consensus.cc:348] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.126806 7864 raft_consensus.cc:374] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.126847 7864 raft_consensus.cc:729] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.126987 7864 consensus_queue.cc:260] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.127188 7866 tablet_bootstrap.cc:492] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:28.127259 7866 ts_tablet_manager.cc:1367] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:28.127293 7864 ts_tablet_manager.cc:1398] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:58:28.127811 7866 raft_consensus.cc:348] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.127918 7866 raft_consensus.cc:374] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.127961 7866 raft_consensus.cc:729] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.128064 7866 consensus_queue.cc:260] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.128321 7866 ts_tablet_manager.cc:1398] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.129094 7863 tablet_bootstrap.cc:654] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.129822 7862 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:28.129904 7862 ts_tablet_manager.cc:1367] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:58:28.130465 7862 raft_consensus.cc:348] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.130568 7862 raft_consensus.cc:374] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.130625 7862 raft_consensus.cc:729] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.130760 7862 consensus_queue.cc:260] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.131170 7862 ts_tablet_manager.cc:1398] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:28.131374 7863 tablet_bootstrap.cc:492] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:28.131465 7863 ts_tablet_manager.cc:1367] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:28.131878 7863 raft_consensus.cc:348] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.132005 7863 raft_consensus.cc:374] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.132058 7863 raft_consensus.cc:729] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.132187 7863 consensus_queue.cc:260] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.132467 7863 ts_tablet_manager.cc:1398] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:28.132632 7863 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:28.133575 7863 tablet_bootstrap.cc:654] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.134637 7863 tablet_bootstrap.cc:492] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:28.134725 7863 ts_tablet_manager.cc:1367] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.135115 7863 raft_consensus.cc:348] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.135201 7863 raft_consensus.cc:374] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.135233 7863 raft_consensus.cc:729] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.135352 7863 consensus_queue.cc:260] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.135556 7863 ts_tablet_manager.cc:1398] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:58:28.192381 7057 tablet.cc:2369] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:28.226394 7873 raft_consensus.cc:482] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.226593 7873 raft_consensus.cc:504] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.227780 7873 leader_election.cc:288] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.229437 7873 raft_consensus.cc:482] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.229547 7873 raft_consensus.cc:504] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.229952 7873 leader_election.cc:288] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.232514 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc04acac844b5ab0995fad69e51bbe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:28.232813 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc04acac844b5ab0995fad69e51bbe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:28.232860 7010 raft_consensus.cc:2454] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.233073 6879 raft_consensus.cc:2454] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.233188 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7033ccd32d43f49df11d96a8752276" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:28.233276 6878 raft_consensus.cc:2454] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.233336 7208 leader_election.cc:302] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:28.233553 7873 raft_consensus.cc:2781] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.233562 7207 leader_election.cc:302] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:28.233637 7873 raft_consensus.cc:482] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.233686 7873 raft_consensus.cc:3037] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.233983 7882 raft_consensus.cc:2781] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.234051 7882 raft_consensus.cc:482] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.234153 7882 raft_consensus.cc:3037] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.234874 7873 raft_consensus.cc:504] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.234930 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7033ccd32d43f49df11d96a8752276" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:28.234987 7882 raft_consensus.cc:504] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.235234 7141 raft_consensus.cc:2454] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.235276 7873 leader_election.cc:288] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.235610 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7033ccd32d43f49df11d96a8752276" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:28.235657 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc04acac844b5ab0995fad69e51bbe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:28.235710 7141 raft_consensus.cc:3037] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.235774 6878 raft_consensus.cc:3037] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.236054 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7033ccd32d43f49df11d96a8752276" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:28.236140 6879 raft_consensus.cc:3037] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.236536 7141 raft_consensus.cc:2454] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.236943 6878 raft_consensus.cc:2454] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.236999 6879 raft_consensus.cc:2454] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.235287 7882 leader_election.cc:288] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.236833 7206 leader_election.cc:302] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.237381 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bc04acac844b5ab0995fad69e51bbe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:28.237425 7882 raft_consensus.cc:2781] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.237495 7010 raft_consensus.cc:3037] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.237515 7207 leader_election.cc:302] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:28.237519 7882 raft_consensus.cc:686] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:28.237684 7873 raft_consensus.cc:2781] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.237892 7873 raft_consensus.cc:686] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:28.237895 7882 consensus_queue.cc:237] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.238026 7873 consensus_queue.cc:237] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:28.238605 7010 raft_consensus.cc:2454] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.240823 6737 catalog_manager.cc:5484] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:28.243927 6737 catalog_manager.cc:5484] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:28.269168 6926 tablet.cc:2369] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:28.293732 7319 tablet.cc:2369] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:28.329861 7872 raft_consensus.cc:482] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.330006 7872 raft_consensus.cc:504] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.331137 7872 leader_election.cc:288] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.334928 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b6f304fb9141fea296a48d35eb33ef" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:28.335109 7141 raft_consensus.cc:2454] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:28.335175 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b6f304fb9141fea296a48d35eb33ef" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:28.335309 6879 raft_consensus.cc:2454] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:28.335470 6944 leader_election.cc:302] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.335721 7872 raft_consensus.cc:2781] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.335789 7872 raft_consensus.cc:482] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.335817 7872 raft_consensus.cc:3037] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.336648 7872 raft_consensus.cc:504] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.336985 7872 leader_election.cc:288] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:28.337291 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b6f304fb9141fea296a48d35eb33ef" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:28.337302 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b6f304fb9141fea296a48d35eb33ef" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:28.337391 7141 raft_consensus.cc:3037] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.337395 6879 raft_consensus.cc:3037] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.338101 7141 raft_consensus.cc:2454] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:28.338109 6879 raft_consensus.cc:2454] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:28.338357 6944 leader_election.cc:302] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.338551 7872 raft_consensus.cc:2781] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.338737 7872 raft_consensus.cc:686] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:28.339061 7872 consensus_queue.cc:237] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:28.341740 6737 catalog_manager.cc:5484] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
W20240501 07:58:28.369462 7188 tablet.cc:2369] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:28.489934 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:28.490175 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:28.493139 7890 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:58:28.580049 7010 raft_consensus.cc:1261] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.580085 6879 raft_consensus.cc:1261] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.580747 6879 raft_consensus.cc:1261] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.580813 7141 raft_consensus.cc:1261] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.580895 6878 raft_consensus.cc:1261] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.581318 7872 consensus_queue.cc:1035] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.581460 7882 consensus_queue.cc:1035] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.581573 7888 consensus_queue.cc:1035] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.582078 7882 consensus_queue.cc:1035] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.582083 7873 consensus_queue.cc:1035] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.583267 7140 raft_consensus.cc:1261] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:28.583631 7873 consensus_queue.cc:1035] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.590860 7921 mvcc.cc:205] Tried to move back new op lower bound from 7022798063942377472 to 7022798062545006592. Current Snapshot: MvccSnapshot[applied={T|T < 7022798063942377472}]
I20240501 07:58:28.599301 7925 mvcc.cc:205] Tried to move back new op lower bound from 7022798063937699840 to 7022798062545092608. Current Snapshot: MvccSnapshot[applied={T|T < 7022798063937699840}]
I20240501 07:58:28.600226 7919 mvcc.cc:205] Tried to move back new op lower bound from 7022798063985680384 to 7022798062959247360. Current Snapshot: MvccSnapshot[applied={T|T < 7022798063938453504}]
I20240501 07:58:28.609817 7931 mvcc.cc:205] Tried to move back new op lower bound from 7022798063938453504 to 7022798062959247360. Current Snapshot: MvccSnapshot[applied={T|T < 7022798063938453504}]
INSERT report
rows total: 2000
time total: 108.581 ms
time per row: 0.0542907 ms
I20240501 07:58:28.648689 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:28.653999 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 9051ba90bad3430fa2e128c88ec7b6f4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.654083 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 9051ba90bad3430fa2e128c88ec7b6f4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.654338 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051ba90bad3430fa2e128c88ec7b6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.654388 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051ba90bad3430fa2e128c88ec7b6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.654886 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 9051ba90bad3430fa2e128c88ec7b6f4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.655098 6859 tablet_service.cc:1467] Processing CreateTablet for tablet c2fa7739592145d5b90d98fecdbe02b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.655175 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051ba90bad3430fa2e128c88ec7b6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.655257 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 93af29aad2ff498dbd8bccaba12f6a70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.655359 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fa7739592145d5b90d98fecdbe02b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.655475 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93af29aad2ff498dbd8bccaba12f6a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.657230 6990 tablet_service.cc:1467] Processing CreateTablet for tablet c2fa7739592145d5b90d98fecdbe02b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.657449 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fa7739592145d5b90d98fecdbe02b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.658154 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 93af29aad2ff498dbd8bccaba12f6a70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.658255 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c2fa7739592145d5b90d98fecdbe02b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.658383 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93af29aad2ff498dbd8bccaba12f6a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.658466 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fa7739592145d5b90d98fecdbe02b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.659286 7951 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:28.660154 7954 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:28.660272 7951 tablet_bootstrap.cc:654] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.660686 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 93af29aad2ff498dbd8bccaba12f6a70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.660923 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93af29aad2ff498dbd8bccaba12f6a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.661121 7954 tablet_bootstrap.cc:654] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.661406 7953 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:28.661492 7951 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:28.661592 7951 ts_tablet_manager.cc:1367] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.661721 7952 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:28.662087 7954 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:28.662082 7951 raft_consensus.cc:348] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.662180 7954 ts_tablet_manager.cc:1367] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.662194 7951 raft_consensus.cc:374] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.662230 7951 raft_consensus.cc:729] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.662360 7951 consensus_queue.cc:260] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.662658 7951 ts_tablet_manager.cc:1398] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.662617 7954 raft_consensus.cc:348] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.662717 7954 raft_consensus.cc:374] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.662755 7954 raft_consensus.cc:729] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.662855 7951 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:28.662904 7954 consensus_queue.cc:260] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.663429 7952 tablet_bootstrap.cc:654] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.663652 7954 ts_tablet_manager.cc:1398] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.663697 7951 tablet_bootstrap.cc:654] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.663877 7954 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:28.664402 7953 tablet_bootstrap.cc:654] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.664745 7954 tablet_bootstrap.cc:654] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.664757 7951 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:28.664932 7951 ts_tablet_manager.cc:1367] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:28.665361 7951 raft_consensus.cc:348] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.665771 7951 raft_consensus.cc:374] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.665799 7954 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:28.665894 7954 ts_tablet_manager.cc:1367] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.665875 7951 raft_consensus.cc:729] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.666123 7952 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:28.666090 7951 consensus_queue.cc:260] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.666208 7952 ts_tablet_manager.cc:1367] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:28.666342 7954 raft_consensus.cc:348] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.666448 7954 raft_consensus.cc:374] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.666486 7954 raft_consensus.cc:729] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.666575 7951 ts_tablet_manager.cc:1398] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.666615 7954 consensus_queue.cc:260] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.666662 7952 raft_consensus.cc:348] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.666766 7952 raft_consensus.cc:374] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.666811 7952 raft_consensus.cc:729] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.666918 7954 ts_tablet_manager.cc:1398] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.666940 7951 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:28.666968 7952 consensus_queue.cc:260] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.667353 7952 ts_tablet_manager.cc:1398] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.667534 7952 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:28.668020 7951 tablet_bootstrap.cc:654] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.668417 7952 tablet_bootstrap.cc:654] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.669600 7953 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:28.669693 7953 ts_tablet_manager.cc:1367] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:28.669868 7952 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:28.669965 7952 ts_tablet_manager.cc:1367] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:28.670174 7951 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:28.670176 7953 raft_consensus.cc:348] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.670270 7951 ts_tablet_manager.cc:1367] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:28.670291 7953 raft_consensus.cc:374] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.670326 7953 raft_consensus.cc:729] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.670382 7952 raft_consensus.cc:348] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.670462 7953 consensus_queue.cc:260] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.670521 7952 raft_consensus.cc:374] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.670576 7952 raft_consensus.cc:729] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.670727 7953 ts_tablet_manager.cc:1398] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:28.670704 7952 consensus_queue.cc:260] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.670886 7951 raft_consensus.cc:348] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.671033 7951 raft_consensus.cc:374] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.671087 7951 raft_consensus.cc:729] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.671214 7951 consensus_queue.cc:260] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.671470 7951 ts_tablet_manager.cc:1398] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:28.671622 7952 ts_tablet_manager.cc:1398] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:28.671797 7952 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:28.672923 7952 tablet_bootstrap.cc:654] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.673970 7952 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:28.674046 7952 ts_tablet_manager.cc:1367] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:28.674472 7952 raft_consensus.cc:348] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.674587 7952 raft_consensus.cc:374] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.674619 7952 raft_consensus.cc:729] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.674736 7952 consensus_queue.cc:260] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.674993 7952 ts_tablet_manager.cc:1398] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:28.684298 7870 raft_consensus.cc:482] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.684417 7870 raft_consensus.cc:504] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.685510 7870 leader_election.cc:288] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:28.689289 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fa7739592145d5b90d98fecdbe02b2" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:28.689324 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fa7739592145d5b90d98fecdbe02b2" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:28.689456 7010 raft_consensus.cc:2454] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:28.689456 6877 raft_consensus.cc:2454] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:28.689818 7076 leader_election.cc:302] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.690063 7870 raft_consensus.cc:2781] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.690124 7870 raft_consensus.cc:482] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.690177 7870 raft_consensus.cc:3037] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.690958 7870 raft_consensus.cc:504] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.691298 7870 leader_election.cc:288] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:28.691632 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fa7739592145d5b90d98fecdbe02b2" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:28.691723 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fa7739592145d5b90d98fecdbe02b2" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:28.691764 7010 raft_consensus.cc:3037] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.691841 6877 raft_consensus.cc:3037] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.692739 6877 raft_consensus.cc:2454] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:28.692739 7010 raft_consensus.cc:2454] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:28.693033 7077 leader_election.cc:302] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.693297 7870 raft_consensus.cc:2781] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.693504 7870 raft_consensus.cc:686] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:28.693825 7870 consensus_queue.cc:237] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.695724 6737 catalog_manager.cc:5484] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:28.722919 7873 raft_consensus.cc:482] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.723099 7873 raft_consensus.cc:504] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.723541 7873 leader_election.cc:288] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:28.724028 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051ba90bad3430fa2e128c88ec7b6f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:28.724028 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051ba90bad3430fa2e128c88ec7b6f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:28.724174 7140 raft_consensus.cc:2454] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.724174 7010 raft_consensus.cc:2454] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:28.724493 7206 leader_election.cc:302] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:28.724843 7873 raft_consensus.cc:2781] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.724911 7873 raft_consensus.cc:482] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.724959 7873 raft_consensus.cc:3037] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.725916 7873 raft_consensus.cc:504] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.726220 7873 leader_election.cc:288] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:28.726583 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051ba90bad3430fa2e128c88ec7b6f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:28.726553 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051ba90bad3430fa2e128c88ec7b6f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:28.726712 7010 raft_consensus.cc:3037] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.726711 7140 raft_consensus.cc:3037] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.727742 7140 raft_consensus.cc:2454] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.727742 7010 raft_consensus.cc:2454] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:28.728057 7208 leader_election.cc:302] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:28.728299 7873 raft_consensus.cc:2781] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.728386 7873 raft_consensus.cc:686] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:28.728528 7873 consensus_queue.cc:237] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:28.730046 6737 catalog_manager.cc:5484] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:29.000119 7872 raft_consensus.cc:482] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.000252 7872 raft_consensus.cc:504] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.000746 7872 leader_election.cc:288] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:29.001549 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93af29aad2ff498dbd8bccaba12f6a70" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:29.001708 7140 raft_consensus.cc:2454] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:29.002017 6944 leader_election.cc:302] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.002700 7872 raft_consensus.cc:2781] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.002753 7872 raft_consensus.cc:482] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.002779 7872 raft_consensus.cc:3037] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.003671 7872 raft_consensus.cc:504] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.004025 7872 leader_election.cc:288] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:29.004387 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93af29aad2ff498dbd8bccaba12f6a70" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:29.004513 7140 raft_consensus.cc:3037] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.004453 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93af29aad2ff498dbd8bccaba12f6a70" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:29.004453 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93af29aad2ff498dbd8bccaba12f6a70" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:29.004695 7272 raft_consensus.cc:2454] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:29.005373 7140 raft_consensus.cc:2454] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:29.005647 6944 leader_election.cc:302] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters: 15ab96f6477b4291932409b80af07213
I20240501 07:58:29.005864 7872 raft_consensus.cc:2781] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.005954 7872 raft_consensus.cc:686] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:29.006096 7872 consensus_queue.cc:237] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.007917 6737 catalog_manager.cc:5484] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:29.121181 7873 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.124531 7882 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:58:29.132018 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:29.132282 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:29.135480 7971 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:29.136104 7870 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.139566 7966 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 07:58:29.239905 7271 raft_consensus.cc:3037] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.241163 7271 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.241710 7140 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.241945 7888 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.242954 7888 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.267 ms
time per row: 0.0626334 ms
I20240501 07:58:29.305423 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:29.310797 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 58dc740b379142029e23f32382abc7b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.310909 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 58dc740b379142029e23f32382abc7b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.311177 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58dc740b379142029e23f32382abc7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.311209 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58dc740b379142029e23f32382abc7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.311908 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.312124 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.312070 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.312336 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.313472 8025 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:29.313781 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 58dc740b379142029e23f32382abc7b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.314039 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58dc740b379142029e23f32382abc7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.314008 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.314203 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017dd2787fba49b7aca26fe954d8fab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.314471 8025 tablet_bootstrap.cc:654] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.315171 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.315218 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.315390 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.315682 8024 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:29.316174 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.316375 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017dd2787fba49b7aca26fe954d8fab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.316776 8024 tablet_bootstrap.cc:654] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.317477 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 017dd2787fba49b7aca26fe954d8fab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.317494 8028 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:29.318324 8028 tablet_bootstrap.cc:654] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.318671 8026 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:29.319489 8024 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:29.319840 8026 tablet_bootstrap.cc:654] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.319952 8025 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:29.319983 8028 tablet_bootstrap.cc:492] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:29.320047 8028 ts_tablet_manager.cc:1367] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:29.320044 8025 ts_tablet_manager.cc:1367] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:29.320442 8028 raft_consensus.cc:348] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.320442 8025 raft_consensus.cc:348] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.320566 8028 raft_consensus.cc:374] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.320566 8025 raft_consensus.cc:374] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.320606 8028 raft_consensus.cc:729] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.320606 8025 raft_consensus.cc:729] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.320736 8025 consensus_queue.cc:260] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.320755 8028 consensus_queue.cc:260] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.321020 8025 ts_tablet_manager.cc:1398] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.321069 8026 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:29.321089 8028 ts_tablet_manager.cc:1398] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.321144 8026 ts_tablet_manager.cc:1367] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:29.321184 8025 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:29.321625 8028 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:29.321588 8026 raft_consensus.cc:348] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.321700 8026 raft_consensus.cc:374] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.321741 8026 raft_consensus.cc:729] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.321907 8026 consensus_queue.cc:260] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.322031 8025 tablet_bootstrap.cc:654] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.322266 8026 ts_tablet_manager.cc:1398] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.322751 8028 tablet_bootstrap.cc:654] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.323119 8025 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:29.323204 8025 ts_tablet_manager.cc:1367] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:29.323786 8025 raft_consensus.cc:348] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.323882 8025 raft_consensus.cc:374] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.323913 8025 raft_consensus.cc:729] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.324080 8025 consensus_queue.cc:260] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.324136 8028 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:29.324210 8028 ts_tablet_manager.cc:1367] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:29.324635 8028 raft_consensus.cc:348] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.324707 8024 ts_tablet_manager.cc:1367] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:29.324731 8028 raft_consensus.cc:374] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.324762 8028 raft_consensus.cc:729] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.324873 8028 consensus_queue.cc:260] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.325157 8028 ts_tablet_manager.cc:1398] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:29.325299 8024 raft_consensus.cc:348] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.325404 8024 raft_consensus.cc:374] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.325438 8024 raft_consensus.cc:729] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.325590 8024 consensus_queue.cc:260] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.325829 8025 ts_tablet_manager.cc:1398] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:29.325980 8025 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:29.326277 8024 ts_tablet_manager.cc:1398] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:29.326472 8024 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:29.327064 8025 tablet_bootstrap.cc:654] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.327487 8024 tablet_bootstrap.cc:654] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.328161 8025 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:29.328243 8025 ts_tablet_manager.cc:1367] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:29.328554 8024 tablet_bootstrap.cc:492] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:29.328639 8024 ts_tablet_manager.cc:1367] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:29.328649 8025 raft_consensus.cc:348] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.328765 8025 raft_consensus.cc:374] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.328815 8025 raft_consensus.cc:729] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.328927 8025 consensus_queue.cc:260] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.329110 8024 raft_consensus.cc:348] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.329202 8024 raft_consensus.cc:374] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.329207 8025 ts_tablet_manager.cc:1398] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:29.329243 8024 raft_consensus.cc:729] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.329363 8024 consensus_queue.cc:260] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.329617 8024 ts_tablet_manager.cc:1398] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:29.329789 8024 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:29.330641 8024 tablet_bootstrap.cc:654] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.331717 8024 tablet_bootstrap.cc:492] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:29.331802 8024 ts_tablet_manager.cc:1367] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:29.332123 8024 raft_consensus.cc:348] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.332214 8024 raft_consensus.cc:374] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.332255 8024 raft_consensus.cc:729] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.332393 8024 consensus_queue.cc:260] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.332638 8024 ts_tablet_manager.cc:1398] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:29.346802 7872 raft_consensus.cc:482] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.346936 7872 raft_consensus.cc:504] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.347427 7872 leader_election.cc:288] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.347780 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:29.347921 6877 raft_consensus.cc:2454] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:29.347923 7974 raft_consensus.cc:482] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.347962 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:29.348004 7974 raft_consensus.cc:504] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.348163 7271 raft_consensus.cc:2454] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:29.348318 6945 leader_election.cc:302] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:29.348392 7974 leader_election.cc:288] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.348744 7872 raft_consensus.cc:2781] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.348775 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:29.348824 7872 raft_consensus.cc:482] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.348817 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:29.348877 7872 raft_consensus.cc:3037] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.348915 6877 raft_consensus.cc:2454] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:29.349208 7207 leader_election.cc:302] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:29.349378 7974 raft_consensus.cc:2781] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.349431 7974 raft_consensus.cc:482] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.349464 7974 raft_consensus.cc:3037] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.349884 7872 raft_consensus.cc:504] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.350178 7009 raft_consensus.cc:2379] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15ab96f6477b4291932409b80af07213 in current term 1: Already voted for candidate 9a3888360d4d4ace96b6b909c160bfc2 in this term.
I20240501 07:58:29.350227 7872 leader_election.cc:288] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.350231 7974 raft_consensus.cc:504] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.350587 7974 leader_election.cc:288] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.350642 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:29.350756 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:29.350800 7271 raft_consensus.cc:2379] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3888360d4d4ace96b6b909c160bfc2 in current term 1: Already voted for candidate 15ab96f6477b4291932409b80af07213 in this term.
I20240501 07:58:29.350847 6877 raft_consensus.cc:3037] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.351035 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:29.351328 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "017dd2787fba49b7aca26fe954d8fab4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:29.351455 7009 raft_consensus.cc:2379] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 15ab96f6477b4291932409b80af07213 in current term 1: Already voted for candidate 9a3888360d4d4ace96b6b909c160bfc2 in this term.
I20240501 07:58:29.351814 7208 leader_election.cc:302] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15ab96f6477b4291932409b80af07213; no voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:58:29.351825 6877 raft_consensus.cc:2454] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:29.352032 7974 raft_consensus.cc:2726] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 07:58:29.352093 6945 leader_election.cc:302] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters: 15ab96f6477b4291932409b80af07213
I20240501 07:58:29.352277 7872 raft_consensus.cc:2781] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.352357 7872 raft_consensus.cc:686] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:29.352497 7872 consensus_queue.cc:237] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.354079 6732 catalog_manager.cc:5484] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:29.430692 7974 raft_consensus.cc:482] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.430809 7974 raft_consensus.cc:504] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.431166 7974 leader_election.cc:288] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.431635 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58dc740b379142029e23f32382abc7b3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:29.431668 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58dc740b379142029e23f32382abc7b3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:29.431759 7009 raft_consensus.cc:2454] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:29.431835 6877 raft_consensus.cc:2454] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:29.432039 7208 leader_election.cc:302] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:29.432190 7974 raft_consensus.cc:2781] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.432250 7974 raft_consensus.cc:482] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.432286 7974 raft_consensus.cc:3037] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.433071 7974 raft_consensus.cc:504] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.433346 7974 leader_election.cc:288] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.433712 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58dc740b379142029e23f32382abc7b3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:29.433770 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58dc740b379142029e23f32382abc7b3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:29.433847 7009 raft_consensus.cc:3037] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.433878 6877 raft_consensus.cc:3037] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.434633 6877 raft_consensus.cc:2454] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:29.434809 7009 raft_consensus.cc:2454] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:29.434955 7207 leader_election.cc:302] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:29.435204 7974 raft_consensus.cc:2781] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.435278 7974 raft_consensus.cc:686] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:29.435428 7974 consensus_queue.cc:237] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.437077 6732 catalog_manager.cc:5484] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:29.440728 7922 raft_consensus.cc:482] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.440834 7922 raft_consensus.cc:504] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.441833 7922 leader_election.cc:288] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.446000 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf7cfe95b0c44f480f4bf2dd8c78769" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:29.446132 7271 raft_consensus.cc:2454] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:29.446455 6813 leader_election.cc:302] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:29.446686 7922 raft_consensus.cc:2781] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.446666 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf7cfe95b0c44f480f4bf2dd8c78769" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:29.446749 7922 raft_consensus.cc:482] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.446802 7922 raft_consensus.cc:3037] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.446802 7140 raft_consensus.cc:2454] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:29.447654 7922 raft_consensus.cc:504] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.447954 7922 leader_election.cc:288] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.448308 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf7cfe95b0c44f480f4bf2dd8c78769" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:29.448372 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf7cfe95b0c44f480f4bf2dd8c78769" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:29.448421 7140 raft_consensus.cc:3037] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.448489 7271 raft_consensus.cc:3037] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.449190 7140 raft_consensus.cc:2454] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:29.449466 6813 leader_election.cc:302] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.449687 7922 raft_consensus.cc:2781] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.449748 7271 raft_consensus.cc:2454] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:29.449905 7922 raft_consensus.cc:686] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:29.450206 7922 consensus_queue.cc:237] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:29.452378 6732 catalog_manager.cc:5484] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:29.583061 8043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:29.583326 8043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:29.586342 8043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:58:29.662499 6877 raft_consensus.cc:1261] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.662555 7271 raft_consensus.cc:1261] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.662854 7272 raft_consensus.cc:1261] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.663672 7888 consensus_queue.cc:1035] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.664387 7009 raft_consensus.cc:1261] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.664389 7922 consensus_queue.cc:1035] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.664693 6878 raft_consensus.cc:1261] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.665432 7974 consensus_queue.cc:1035] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.665445 7888 consensus_queue.cc:1035] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.665779 7882 consensus_queue.cc:1035] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:29.667243 7140 raft_consensus.cc:1261] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:29.673264 8041 consensus_queue.cc:1035] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 98.1017 ms
time per row: 0.0490509 ms
I20240501 07:58:29.727778 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:29.733156 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 43b82e035ecb4e23973a2163e15c1dac (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.733161 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 43b82e035ecb4e23973a2163e15c1dac (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.733490 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b82e035ecb4e23973a2163e15c1dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.733511 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b82e035ecb4e23973a2163e15c1dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.733623 7250 tablet_service.cc:1467] Processing CreateTablet for tablet f334b66a84b94c3ba15004253b4cb714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.733886 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f334b66a84b94c3ba15004253b4cb714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.735404 8025 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:29.735769 8026 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:29.736056 8024 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:29.736390 8025 tablet_bootstrap.cc:654] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.736479 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 43b82e035ecb4e23973a2163e15c1dac (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.736709 8026 tablet_bootstrap.cc:654] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.736753 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b82e035ecb4e23973a2163e15c1dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.736907 8024 tablet_bootstrap.cc:654] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.737965 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 80152bb09e4e42a09e106cf8b58178e3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.738193 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80152bb09e4e42a09e106cf8b58178e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.738324 8026 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:29.738399 8026 ts_tablet_manager.cc:1367] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:29.738842 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 80152bb09e4e42a09e106cf8b58178e3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.738838 8026 raft_consensus.cc:348] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.738950 8026 raft_consensus.cc:374] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.738981 8026 raft_consensus.cc:729] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.739075 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80152bb09e4e42a09e106cf8b58178e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.739126 8026 consensus_queue.cc:260] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.739418 8026 ts_tablet_manager.cc:1398] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.740679 8024 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:29.740670 8028 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:29.740760 8024 ts_tablet_manager.cc:1367] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:29.741133 8024 raft_consensus.cc:348] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.741240 8024 raft_consensus.cc:374] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.741282 8024 raft_consensus.cc:729] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.741411 8024 consensus_queue.cc:260] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.741511 8028 tablet_bootstrap.cc:654] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.741689 8024 ts_tablet_manager.cc:1398] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.741859 8024 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:29.742729 8024 tablet_bootstrap.cc:654] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.742730 6858 tablet_service.cc:1467] Processing CreateTablet for tablet f334b66a84b94c3ba15004253b4cb714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.742990 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f334b66a84b94c3ba15004253b4cb714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.743221 7121 tablet_service.cc:1467] Processing CreateTablet for tablet f334b66a84b94c3ba15004253b4cb714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.743340 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 80152bb09e4e42a09e106cf8b58178e3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:29.743444 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f334b66a84b94c3ba15004253b4cb714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.743551 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80152bb09e4e42a09e106cf8b58178e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:29.745397 8026 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:29.745954 8028 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:29.746037 8028 ts_tablet_manager.cc:1367] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:58:29.746073 8024 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:29.746135 8024 ts_tablet_manager.cc:1367] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:29.746255 8026 tablet_bootstrap.cc:654] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.746485 8028 raft_consensus.cc:348] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.746497 8024 raft_consensus.cc:348] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.746584 8028 raft_consensus.cc:374] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.746592 8024 raft_consensus.cc:374] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.746620 8028 raft_consensus.cc:729] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.746626 8024 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.746747 8024 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.746747 8028 consensus_queue.cc:260] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.746999 8028 ts_tablet_manager.cc:1398] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:29.747056 8024 ts_tablet_manager.cc:1398] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:29.747184 8028 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:29.747287 8026 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:29.747356 8026 ts_tablet_manager.cc:1367] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:29.747851 8028 tablet_bootstrap.cc:654] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.747772 8026 raft_consensus.cc:348] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.747906 8026 raft_consensus.cc:374] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.747960 8026 raft_consensus.cc:729] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.748090 8026 consensus_queue.cc:260] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.748361 8026 ts_tablet_manager.cc:1398] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.748502 8026 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:29.749353 8025 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:29.749449 8025 ts_tablet_manager.cc:1367] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 07:58:29.749673 8026 tablet_bootstrap.cc:654] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.749962 8025 raft_consensus.cc:348] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.750072 8025 raft_consensus.cc:374] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.750106 8025 raft_consensus.cc:729] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.750237 8025 consensus_queue.cc:260] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.750495 8025 ts_tablet_manager.cc:1398] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:29.750553 8028 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:29.750635 8028 ts_tablet_manager.cc:1367] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:29.750670 8025 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:29.751111 8028 raft_consensus.cc:348] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.751224 8028 raft_consensus.cc:374] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.751259 8028 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.751377 8028 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.751578 8025 tablet_bootstrap.cc:654] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:29.751672 8028 ts_tablet_manager.cc:1398] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.751941 8026 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:29.752032 8026 ts_tablet_manager.cc:1367] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:29.752451 8026 raft_consensus.cc:348] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.752569 8026 raft_consensus.cc:374] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.752612 8026 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.752727 8026 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.753002 8026 ts_tablet_manager.cc:1398] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.754493 8025 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:29.754576 8025 ts_tablet_manager.cc:1367] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:29.754949 8025 raft_consensus.cc:348] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.755088 8025 raft_consensus.cc:374] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:29.755138 8025 raft_consensus.cc:729] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:29.755262 8025 consensus_queue.cc:260] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.755841 8025 ts_tablet_manager.cc:1398] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:29.758693 8084 raft_consensus.cc:482] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.758769 8084 raft_consensus.cc:504] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.759150 8084 leader_election.cc:288] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.759441 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b82e035ecb4e23973a2163e15c1dac" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:29.759580 7140 raft_consensus.cc:2454] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:29.759829 6813 leader_election.cc:302] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.759985 8084 raft_consensus.cc:2781] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.760037 8084 raft_consensus.cc:482] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.760087 8084 raft_consensus.cc:3037] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.760857 8084 raft_consensus.cc:504] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.761163 8084 leader_election.cc:288] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.761457 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b82e035ecb4e23973a2163e15c1dac" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:29.761570 7140 raft_consensus.cc:3037] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.762311 7140 raft_consensus.cc:2454] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:29.762562 6813 leader_election.cc:302] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.762720 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b82e035ecb4e23973a2163e15c1dac" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:29.762741 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b82e035ecb4e23973a2163e15c1dac" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:29.762866 7009 raft_consensus.cc:2454] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:29.763043 8084 raft_consensus.cc:2781] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.763127 8084 raft_consensus.cc:686] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:29.763249 8084 consensus_queue.cc:237] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:29.764834 6733 catalog_manager.cc:5484] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:29.799906 8011 raft_consensus.cc:482] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.800050 8011 raft_consensus.cc:504] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.800590 8011 leader_election.cc:288] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.801039 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f334b66a84b94c3ba15004253b4cb714" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:29.801188 6877 raft_consensus.cc:2454] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:29.801483 7076 leader_election.cc:302] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.801669 8011 raft_consensus.cc:2781] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.801728 8011 raft_consensus.cc:482] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.801774 8011 raft_consensus.cc:3037] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.802793 8011 raft_consensus.cc:504] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.803211 8011 leader_election.cc:288] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:29.803488 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f334b66a84b94c3ba15004253b4cb714" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:29.803619 6877 raft_consensus.cc:3037] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.804446 6877 raft_consensus.cc:2454] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:29.804737 7076 leader_election.cc:302] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.804733 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f334b66a84b94c3ba15004253b4cb714" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:29.804780 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f334b66a84b94c3ba15004253b4cb714" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:29.804880 7272 raft_consensus.cc:2454] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:29.804916 8011 raft_consensus.cc:2781] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.805074 8011 raft_consensus.cc:686] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:29.805219 8011 consensus_queue.cc:237] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:29.806833 6733 catalog_manager.cc:5484] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:29.837397 8011 raft_consensus.cc:482] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:29.837550 8011 raft_consensus.cc:504] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.837918 8011 leader_election.cc:288] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.838320 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:29.838361 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:29.838449 7271 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:29.838526 7009 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:29.838747 7075 leader_election.cc:302] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.838941 8011 raft_consensus.cc:2781] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:29.838997 8011 raft_consensus.cc:482] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:29.839073 8011 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.839890 8011 raft_consensus.cc:504] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.840341 8011 leader_election.cc:288] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:29.840534 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:29.840539 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:29.840652 7009 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.840660 7271 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:29.841398 7271 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:29.841416 7009 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:29.841730 7077 leader_election.cc:302] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:29.841948 8011 raft_consensus.cc:2781] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:29.842037 8011 raft_consensus.cc:686] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:29.842361 8011 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:29.843917 6733 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
W20240501 07:58:29.971879 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:29.972144 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:29.975217 8101 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:58:30.076165 7009 raft_consensus.cc:3037] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.077152 7009 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.077811 7140 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.078217 8041 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.078384 7922 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.082239 7010 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.082260 6877 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.082986 8098 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.083312 8098 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.085275 7272 raft_consensus.cc:3037] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.085279 7271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.085731 8098 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.086697 7272 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.087136 8011 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.775 ms
time per row: 0.0648876 ms
I20240501 07:58:30.148093 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:30.153474 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.153509 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.153509 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.153795 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.153795 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.153841 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.155858 8025 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:30.155920 8028 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:30.156036 8026 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:30.156803 8025 tablet_bootstrap.cc:654] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.156806 8026 tablet_bootstrap.cc:654] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.156829 8028 tablet_bootstrap.cc:654] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.157439 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 74c1f8c60d0344c381aebc78f198944d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.157675 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c1f8c60d0344c381aebc78f198944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.157985 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 74c1f8c60d0344c381aebc78f198944d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.158182 6990 tablet_service.cc:1467] Processing CreateTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.158267 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c1f8c60d0344c381aebc78f198944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.158394 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5df0c0e783e491e8b37dd349df4ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.158479 6858 tablet_service.cc:1467] Processing CreateTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.158707 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5df0c0e783e491e8b37dd349df4ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.159168 7250 tablet_service.cc:1467] Processing CreateTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.159402 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5df0c0e783e491e8b37dd349df4ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.160650 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 74c1f8c60d0344c381aebc78f198944d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.160876 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c1f8c60d0344c381aebc78f198944d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.160986 8025 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:30.161072 8025 ts_tablet_manager.cc:1367] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:30.161262 8028 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:30.161334 8028 ts_tablet_manager.cc:1367] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:30.161458 8024 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:30.161563 8025 raft_consensus.cc:348] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.161861 8025 raft_consensus.cc:374] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.161911 8025 raft_consensus.cc:729] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.162067 8028 raft_consensus.cc:348] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.162185 8028 raft_consensus.cc:374] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.162223 8028 raft_consensus.cc:729] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.162168 8025 consensus_queue.cc:260] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.162357 8028 consensus_queue.cc:260] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.162608 8028 ts_tablet_manager.cc:1398] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.162755 8028 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:30.162683 8025 ts_tablet_manager.cc:1398] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:30.163529 8025 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:30.163663 8028 tablet_bootstrap.cc:654] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.164465 8025 tablet_bootstrap.cc:654] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.165267 8028 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:30.165364 8028 ts_tablet_manager.cc:1367] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:30.165884 8028 raft_consensus.cc:348] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.166018 8028 raft_consensus.cc:374] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.166045 8028 raft_consensus.cc:729] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.166193 8028 consensus_queue.cc:260] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.166569 8028 ts_tablet_manager.cc:1398] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.168655 8024 tablet_bootstrap.cc:654] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.170203 8025 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:30.170284 8025 ts_tablet_manager.cc:1367] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 07:58:30.170724 8025 raft_consensus.cc:348] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.170866 8025 raft_consensus.cc:374] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.170902 8025 raft_consensus.cc:729] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.171036 8025 consensus_queue.cc:260] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.171504 8025 ts_tablet_manager.cc:1398] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.172402 8024 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:30.172500 8024 ts_tablet_manager.cc:1367] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 07:58:30.173008 8024 raft_consensus.cc:348] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.173136 8024 raft_consensus.cc:374] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.173166 8024 raft_consensus.cc:729] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.173317 8024 consensus_queue.cc:260] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.173619 8024 ts_tablet_manager.cc:1398] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.173797 8024 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:30.173902 8026 tablet_bootstrap.cc:492] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:30.173982 8026 ts_tablet_manager.cc:1367] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:58:30.174425 8026 raft_consensus.cc:348] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.174554 8026 raft_consensus.cc:374] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.174602 8026 raft_consensus.cc:729] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.174695 8024 tablet_bootstrap.cc:654] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.174752 8026 consensus_queue.cc:260] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.175189 8026 ts_tablet_manager.cc:1398] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:30.175359 8026 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:30.175787 8024 tablet_bootstrap.cc:492] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:30.175865 8024 ts_tablet_manager.cc:1367] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:30.176237 8024 raft_consensus.cc:348] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.176331 8026 tablet_bootstrap.cc:654] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.176332 8024 raft_consensus.cc:374] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.176365 8024 raft_consensus.cc:729] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.176505 8024 consensus_queue.cc:260] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.176853 8024 ts_tablet_manager.cc:1398] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:30.177466 8025 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:30.178463 8025 tablet_bootstrap.cc:654] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.179535 8025 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:30.179540 8026 tablet_bootstrap.cc:492] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:30.179620 8025 ts_tablet_manager.cc:1367] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:30.179621 8026 ts_tablet_manager.cc:1367] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:30.180022 8025 raft_consensus.cc:348] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.180048 8026 raft_consensus.cc:348] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.180132 8025 raft_consensus.cc:374] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.180131 8026 raft_consensus.cc:374] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.180171 8026 raft_consensus.cc:729] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.180171 8025 raft_consensus.cc:729] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.180290 8026 consensus_queue.cc:260] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.180290 8025 consensus_queue.cc:260] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.180545 8026 ts_tablet_manager.cc:1398] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.180938 8025 ts_tablet_manager.cc:1398] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.207592 8080 raft_consensus.cc:482] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:30.207760 8080 raft_consensus.cc:504] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.208158 8080 leader_election.cc:288] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.208518 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5df0c0e783e491e8b37dd349df4ad34" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:30.208529 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5df0c0e783e491e8b37dd349df4ad34" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:30.208665 6877 raft_consensus.cc:2454] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:30.208676 7271 raft_consensus.cc:2454] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:30.209009 6944 leader_election.cc:302] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:30.209209 8080 raft_consensus.cc:2781] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:30.209261 8080 raft_consensus.cc:482] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:30.209292 8080 raft_consensus.cc:3037] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.210358 8080 raft_consensus.cc:504] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.210680 8080 leader_election.cc:288] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.210999 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5df0c0e783e491e8b37dd349df4ad34" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:30.211150 7271 raft_consensus.cc:3037] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.211139 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5df0c0e783e491e8b37dd349df4ad34" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:30.211273 6877 raft_consensus.cc:3037] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.212008 6877 raft_consensus.cc:2454] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:30.212015 7271 raft_consensus.cc:2454] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:30.212257 6945 leader_election.cc:302] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:30.212447 8080 raft_consensus.cc:2781] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.212522 8080 raft_consensus.cc:686] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:30.212647 8080 consensus_queue.cc:237] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.214246 6737 catalog_manager.cc:5484] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:30.260462 8098 raft_consensus.cc:482] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:30.260592 8098 raft_consensus.cc:504] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.260984 8098 leader_election.cc:288] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:30.261397 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5dd2edfd7b4e28b226299dc2a4fc04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:30.261399 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5dd2edfd7b4e28b226299dc2a4fc04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:30.261550 6877 raft_consensus.cc:2454] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.261550 7010 raft_consensus.cc:2454] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.261827 7077 leader_election.cc:302] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.261991 8098 raft_consensus.cc:2781] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:30.262044 8098 raft_consensus.cc:482] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:30.262071 8098 raft_consensus.cc:3037] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.262960 8098 raft_consensus.cc:504] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.263293 8098 leader_election.cc:288] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:30.263584 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5dd2edfd7b4e28b226299dc2a4fc04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:30.263602 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5dd2edfd7b4e28b226299dc2a4fc04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:30.263689 7010 raft_consensus.cc:3037] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.263705 6877 raft_consensus.cc:3037] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.264396 7010 raft_consensus.cc:2454] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.264454 6877 raft_consensus.cc:2454] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.264719 7077 leader_election.cc:302] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.264889 8098 raft_consensus.cc:2781] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.264979 8098 raft_consensus.cc:686] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:30.265115 8098 consensus_queue.cc:237] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.266673 6737 catalog_manager.cc:5484] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:30.416152 8098 raft_consensus.cc:482] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:30.416283 8098 raft_consensus.cc:504] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.416680 8098 leader_election.cc:288] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.417110 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c1f8c60d0344c381aebc78f198944d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:30.417131 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c1f8c60d0344c381aebc78f198944d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:30.417246 7271 raft_consensus.cc:2454] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.417266 6877 raft_consensus.cc:2454] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.417558 7075 leader_election.cc:302] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.417709 8098 raft_consensus.cc:2781] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:30.417779 8098 raft_consensus.cc:482] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:30.417830 8098 raft_consensus.cc:3037] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.418658 8098 raft_consensus.cc:504] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.418947 8098 leader_election.cc:288] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.419274 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c1f8c60d0344c381aebc78f198944d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:30.419327 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c1f8c60d0344c381aebc78f198944d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:30.419389 6877 raft_consensus.cc:3037] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.419544 7271 raft_consensus.cc:3037] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.420341 6877 raft_consensus.cc:2454] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.420568 7271 raft_consensus.cc:2454] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.420645 7076 leader_election.cc:302] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.420821 8098 raft_consensus.cc:2781] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.420894 8098 raft_consensus.cc:686] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:30.421041 8098 consensus_queue.cc:237] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.423079 6737 catalog_manager.cc:5484] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:30.572362 8151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:30.572649 8151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:30.575943 8151 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:58:30.633191 8080 consensus_queue.cc:1035] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.637873 8080 consensus_queue.cc:1035] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.670107 7010 raft_consensus.cc:1261] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.670709 8011 consensus_queue.cc:1035] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.670969 6877 raft_consensus.cc:1261] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.671366 7271 raft_consensus.cc:1261] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.671658 8011 consensus_queue.cc:1035] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.672075 8011 consensus_queue.cc:1035] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:30.673025 6879 raft_consensus.cc:1261] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:30.683102 8189 consensus_queue.cc:1035] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.737 ms
time per row: 0.0598683 ms
I20240501 07:58:30.739708 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:30.744887 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c977cb9c5d1b41a29e7671218b917315 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.745013 6990 tablet_service.cc:1467] Processing CreateTablet for tablet c977cb9c5d1b41a29e7671218b917315 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.745241 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977cb9c5d1b41a29e7671218b917315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.745272 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977cb9c5d1b41a29e7671218b917315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.745970 6859 tablet_service.cc:1467] Processing CreateTablet for tablet f723d182d04d439da6ab5b811448cf2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.746233 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f723d182d04d439da6ab5b811448cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.747529 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9379f9bf86403ebf1695263db9a603 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.747754 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9379f9bf86403ebf1695263db9a603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.748026 7250 tablet_service.cc:1467] Processing CreateTablet for tablet c977cb9c5d1b41a29e7671218b917315 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.748319 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977cb9c5d1b41a29e7671218b917315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.748502 7121 tablet_service.cc:1467] Processing CreateTablet for tablet f723d182d04d439da6ab5b811448cf2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.748733 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f723d182d04d439da6ab5b811448cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.748826 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9379f9bf86403ebf1695263db9a603 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.749054 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9379f9bf86403ebf1695263db9a603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.749805 6988 tablet_service.cc:1467] Processing CreateTablet for tablet f723d182d04d439da6ab5b811448cf2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.750020 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f723d182d04d439da6ab5b811448cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.748026 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9379f9bf86403ebf1695263db9a603 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:30.750598 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9379f9bf86403ebf1695263db9a603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.750977 8206 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:30.751156 8209 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:30.751859 8206 tablet_bootstrap.cc:654] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.752389 8209 tablet_bootstrap.cc:654] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.752838 8208 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:30.753067 8207 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:30.753679 8208 tablet_bootstrap.cc:654] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.753858 8207 tablet_bootstrap.cc:654] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.754712 8208 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:30.754798 8208 ts_tablet_manager.cc:1367] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:30.754786 8206 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:30.754882 8206 ts_tablet_manager.cc:1367] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:30.755101 8209 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:30.755183 8209 ts_tablet_manager.cc:1367] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:30.755264 8208 raft_consensus.cc:348] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.755365 8208 raft_consensus.cc:374] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.755405 8208 raft_consensus.cc:729] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.755365 8206 raft_consensus.cc:348] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.755478 8206 raft_consensus.cc:374] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.755515 8206 raft_consensus.cc:729] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.755559 8208 consensus_queue.cc:260] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.755585 8209 raft_consensus.cc:348] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.755640 8206 consensus_queue.cc:260] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.755669 8209 raft_consensus.cc:374] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.755717 8209 raft_consensus.cc:729] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.755827 8209 consensus_queue.cc:260] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.755918 8208 ts_tablet_manager.cc:1398] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.756069 8209 ts_tablet_manager.cc:1398] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.756342 8206 ts_tablet_manager.cc:1398] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:30.756623 8209 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:30.756633 8207 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:30.756705 8207 ts_tablet_manager.cc:1367] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:30.757119 8207 raft_consensus.cc:348] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.757213 8207 raft_consensus.cc:374] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.757247 8207 raft_consensus.cc:729] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.757390 8209 tablet_bootstrap.cc:654] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.757371 8207 consensus_queue.cc:260] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.757642 8207 ts_tablet_manager.cc:1398] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.757812 8207 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:30.758682 8207 tablet_bootstrap.cc:654] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.756520 8206 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:30.760551 8206 tablet_bootstrap.cc:654] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.760624 8207 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:30.760716 8207 ts_tablet_manager.cc:1367] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:30.761153 8207 raft_consensus.cc:348] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.761270 8207 raft_consensus.cc:374] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.761309 8207 raft_consensus.cc:729] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.761432 8207 consensus_queue.cc:260] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.761610 8206 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:30.761688 8206 ts_tablet_manager.cc:1367] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 07:58:30.761735 8207 ts_tablet_manager.cc:1398] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:30.761888 8207 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:30.762089 8206 raft_consensus.cc:348] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.762188 8206 raft_consensus.cc:374] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.762230 8206 raft_consensus.cc:729] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.762364 8206 consensus_queue.cc:260] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.762665 8206 ts_tablet_manager.cc:1398] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:30.762815 8206 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:30.762975 8207 tablet_bootstrap.cc:654] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.763864 8206 tablet_bootstrap.cc:654] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.764573 8207 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:30.764916 8206 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:30.765007 8207 ts_tablet_manager.cc:1367] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:30.765172 8209 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:30.765172 8206 ts_tablet_manager.cc:1367] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:30.765257 8209 ts_tablet_manager.cc:1367] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:58:30.765432 8207 raft_consensus.cc:348] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.765530 8207 raft_consensus.cc:374] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.765574 8207 raft_consensus.cc:729] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.765560 8206 raft_consensus.cc:348] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.765659 8206 raft_consensus.cc:374] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.765698 8206 raft_consensus.cc:729] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.765702 8207 consensus_queue.cc:260] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.765913 8206 consensus_queue.cc:260] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.765966 8207 ts_tablet_manager.cc:1398] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:30.766184 8206 ts_tablet_manager.cc:1398] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:30.766498 8209 raft_consensus.cc:348] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.766592 8209 raft_consensus.cc:374] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.766624 8209 raft_consensus.cc:729] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.766772 8209 consensus_queue.cc:260] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:30.767194 8209 ts_tablet_manager.cc:1398] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:30.791837 8011 raft_consensus.cc:482] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:30.791949 8011 raft_consensus.cc:504] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.792316 8011 leader_election.cc:288] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:30.792712 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f723d182d04d439da6ab5b811448cf2d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:30.792712 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f723d182d04d439da6ab5b811448cf2d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:30.792865 7010 raft_consensus.cc:2454] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.792871 6877 raft_consensus.cc:2454] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.793146 7077 leader_election.cc:302] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.793341 8011 raft_consensus.cc:2781] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:30.793411 8011 raft_consensus.cc:482] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:30.793457 8011 raft_consensus.cc:3037] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.793555 8098 raft_consensus.cc:482] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:30.793635 8098 raft_consensus.cc:504] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.793960 8098 leader_election.cc:288] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.794212 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977cb9c5d1b41a29e7671218b917315" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:30.794332 7010 raft_consensus.cc:2454] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.794342 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977cb9c5d1b41a29e7671218b917315" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:30.794400 8011 raft_consensus.cc:504] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.794493 7271 raft_consensus.cc:2454] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:30.794566 7077 leader_election.cc:302] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.794705 8011 leader_election.cc:288] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:30.794800 8011 raft_consensus.cc:2781] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:30.794842 8011 raft_consensus.cc:482] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:30.794893 8011 raft_consensus.cc:3037] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.795033 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f723d182d04d439da6ab5b811448cf2d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:30.795164 7010 raft_consensus.cc:3037] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.795337 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f723d182d04d439da6ab5b811448cf2d" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:30.795449 6877 raft_consensus.cc:3037] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.795747 8011 raft_consensus.cc:504] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.796166 7010 raft_consensus.cc:2454] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.796370 6877 raft_consensus.cc:2454] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.796404 8011 leader_election.cc:288] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:30.796499 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977cb9c5d1b41a29e7671218b917315" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:30.796600 7077 leader_election.cc:302] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.796619 7271 raft_consensus.cc:3037] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.796765 8011 raft_consensus.cc:2781] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.796851 8011 raft_consensus.cc:686] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:30.796979 8011 consensus_queue.cc:237] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:30.797032 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977cb9c5d1b41a29e7671218b917315" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:30.797134 7010 raft_consensus.cc:3037] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.797632 7271 raft_consensus.cc:2454] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.797955 7075 leader_election.cc:302] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:30.798030 7010 raft_consensus.cc:2454] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:30.798107 8011 raft_consensus.cc:2781] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.798208 8011 raft_consensus.cc:686] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:30.798326 8011 consensus_queue.cc:237] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:30.799687 6733 catalog_manager.cc:5484] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:30.802618 6733 catalog_manager.cc:5484] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:31.075728 8011 raft_consensus.cc:482] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.075850 8011 raft_consensus.cc:504] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.076241 8011 leader_election.cc:288] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:31.076576 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:31.076663 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:31.076757 7271 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:31.076790 7010 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:31.077034 7077 leader_election.cc:302] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.077206 8011 raft_consensus.cc:2781] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.077287 8011 raft_consensus.cc:482] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.077339 8011 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.078342 8011 raft_consensus.cc:504] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.078639 8011 leader_election.cc:288] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:31.078991 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:31.079097 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:31.079156 7010 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.079239 7271 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.080154 7010 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:31.080154 7271 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:31.080421 7077 leader_election.cc:302] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.080593 8011 raft_consensus.cc:2781] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.080682 8011 raft_consensus.cc:686] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:31.080818 8011 consensus_queue.cc:237] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.082378 6733 catalog_manager.cc:5484] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
W20240501 07:58:31.232559 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.232806 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.235905 8222 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 07:58:31.310492 7271 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.310863 7010 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.310952 8098 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.311269 8011 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.320288 7010 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.320828 8098 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.321835 6877 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.322207 8011 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.331069 7009 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.331089 7271 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.331449 8098 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.331563 8011 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.358199 8254 mvcc.cc:205] Tried to move back new op lower bound from 7022798075141656576 to 7022798073031442432. Current Snapshot: MvccSnapshot[applied={T|T < 7022798075141656576}]
INSERT report
rows total: 2000
time total: 97.0469 ms
time per row: 0.0485234 ms
I20240501 07:58:31.377905 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:31.383294 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 0305b131804a4bc096dfe6cd801f98ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.383386 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 0305b131804a4bc096dfe6cd801f98ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.383607 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0305b131804a4bc096dfe6cd801f98ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.383636 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0305b131804a4bc096dfe6cd801f98ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.383989 6858 tablet_service.cc:1467] Processing CreateTablet for tablet fecf2f8d747d4b2694a63034b86635b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.384169 6857 tablet_service.cc:1467] Processing CreateTablet for tablet de03fdd63bd347dea4eb6e5d410da68e (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.384857 6989 tablet_service.cc:1467] Processing CreateTablet for tablet de03fdd63bd347dea4eb6e5d410da68e (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.385030 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 0305b131804a4bc096dfe6cd801f98ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.385123 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de03fdd63bd347dea4eb6e5d410da68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.385274 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0305b131804a4bc096dfe6cd801f98ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.385661 7250 tablet_service.cc:1467] Processing CreateTablet for tablet fecf2f8d747d4b2694a63034b86635b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.385846 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf2f8d747d4b2694a63034b86635b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.386186 7252 tablet_service.cc:1467] Processing CreateTablet for tablet de03fdd63bd347dea4eb6e5d410da68e (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.386365 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de03fdd63bd347dea4eb6e5d410da68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.386787 7120 tablet_service.cc:1467] Processing CreateTablet for tablet fecf2f8d747d4b2694a63034b86635b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.387068 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf2f8d747d4b2694a63034b86635b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.387331 8277 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:31.387809 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de03fdd63bd347dea4eb6e5d410da68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.388260 8277 tablet_bootstrap.cc:654] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.389204 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf2f8d747d4b2694a63034b86635b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.389384 8277 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:31.389470 8277 ts_tablet_manager.cc:1367] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:31.389883 8277 raft_consensus.cc:348] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.389987 8277 raft_consensus.cc:374] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.390023 8277 raft_consensus.cc:729] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.390182 8277 consensus_queue.cc:260] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.391137 8277 ts_tablet_manager.cc:1398] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.391299 8277 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:31.392114 8277 tablet_bootstrap.cc:654] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.388995 8279 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:31.393460 8279 tablet_bootstrap.cc:654] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.394138 8280 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:31.394588 8277 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:31.394665 8277 ts_tablet_manager.cc:1367] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:31.395052 8277 raft_consensus.cc:348] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.395153 8277 raft_consensus.cc:374] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.395190 8277 raft_consensus.cc:729] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.395323 8277 consensus_queue.cc:260] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.395728 8277 ts_tablet_manager.cc:1398] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:31.396461 8280 tablet_bootstrap.cc:654] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.397504 8279 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:31.397585 8279 ts_tablet_manager.cc:1367] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:31.398041 8279 raft_consensus.cc:348] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.398149 8279 raft_consensus.cc:374] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.398191 8279 raft_consensus.cc:729] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.398308 8279 consensus_queue.cc:260] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.398581 8279 ts_tablet_manager.cc:1398] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.399289 8280 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:31.399425 8280 ts_tablet_manager.cc:1367] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:31.399808 8280 raft_consensus.cc:348] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.399919 8280 raft_consensus.cc:374] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.399955 8280 raft_consensus.cc:729] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.400107 8280 consensus_queue.cc:260] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.400439 8280 ts_tablet_manager.cc:1398] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.400660 8280 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:31.401569 8280 tablet_bootstrap.cc:654] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.402885 8280 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:31.402974 8280 ts_tablet_manager.cc:1367] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.403396 8280 raft_consensus.cc:348] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.403486 8280 raft_consensus.cc:374] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.403508 8280 raft_consensus.cc:729] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.403590 8280 consensus_queue.cc:260] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.403803 8280 ts_tablet_manager.cc:1398] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:31.403980 8280 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:31.404830 8280 tablet_bootstrap.cc:654] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.405848 8280 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:31.405951 8280 ts_tablet_manager.cc:1367] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.406296 8280 raft_consensus.cc:348] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.406402 8280 raft_consensus.cc:374] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.406425 8280 raft_consensus.cc:729] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.406517 8280 consensus_queue.cc:260] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.406803 8280 ts_tablet_manager.cc:1398] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.412829 8286 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:31.413601 8286 tablet_bootstrap.cc:654] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.414492 8286 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:31.414561 8286 ts_tablet_manager.cc:1367] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.414892 8286 raft_consensus.cc:348] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.414978 8286 raft_consensus.cc:374] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.415027 8286 raft_consensus.cc:729] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.415155 8286 consensus_queue.cc:260] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.415367 8286 ts_tablet_manager.cc:1398] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:31.415498 8286 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:31.416198 8286 tablet_bootstrap.cc:654] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.417748 8286 tablet_bootstrap.cc:492] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:31.417855 8286 ts_tablet_manager.cc:1367] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:31.418188 8286 raft_consensus.cc:348] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.418298 8286 raft_consensus.cc:374] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.418361 8286 raft_consensus.cc:729] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.418481 8286 consensus_queue.cc:260] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.418711 8286 ts_tablet_manager.cc:1398] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:31.418912 8286 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:31.419662 8286 tablet_bootstrap.cc:654] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.420758 8286 tablet_bootstrap.cc:492] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:31.420847 8286 ts_tablet_manager.cc:1367] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.421270 8286 raft_consensus.cc:348] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.421353 8286 raft_consensus.cc:374] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.421423 8286 raft_consensus.cc:729] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.421547 8286 consensus_queue.cc:260] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.421789 8286 ts_tablet_manager.cc:1398] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.542698 8201 raft_consensus.cc:482] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.542831 8201 raft_consensus.cc:504] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.543308 8201 leader_election.cc:288] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:31.543705 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0305b131804a4bc096dfe6cd801f98ee" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:31.543720 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0305b131804a4bc096dfe6cd801f98ee" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:31.543838 6877 raft_consensus.cc:2454] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:31.543838 7271 raft_consensus.cc:2454] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:31.544150 6944 leader_election.cc:302] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:31.544313 8201 raft_consensus.cc:2781] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.544392 8201 raft_consensus.cc:482] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.544420 8201 raft_consensus.cc:3037] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.545236 8201 raft_consensus.cc:504] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.545558 8201 leader_election.cc:288] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:31.545845 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0305b131804a4bc096dfe6cd801f98ee" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:31.545900 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0305b131804a4bc096dfe6cd801f98ee" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:31.545964 6877 raft_consensus.cc:3037] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.545997 7271 raft_consensus.cc:3037] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.546875 7271 raft_consensus.cc:2454] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:31.546875 6877 raft_consensus.cc:2454] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:31.547196 6944 leader_election.cc:302] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:31.547369 8201 raft_consensus.cc:2781] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.547569 8201 raft_consensus.cc:686] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:31.547742 8201 consensus_queue.cc:237] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.549329 6737 catalog_manager.cc:5484] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:31.562083 8201 raft_consensus.cc:482] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.562188 8201 raft_consensus.cc:504] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.562538 8201 leader_election.cc:288] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:31.562942 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de03fdd63bd347dea4eb6e5d410da68e" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:31.562944 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de03fdd63bd347dea4eb6e5d410da68e" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:31.563091 6877 raft_consensus.cc:2454] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:31.563112 7271 raft_consensus.cc:2454] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:31.563364 6945 leader_election.cc:302] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:31.563524 8201 raft_consensus.cc:2781] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.563588 8201 raft_consensus.cc:482] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.563642 8201 raft_consensus.cc:3037] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.564383 8201 raft_consensus.cc:504] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.564680 8201 leader_election.cc:288] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:31.564949 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de03fdd63bd347dea4eb6e5d410da68e" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:31.565037 6877 raft_consensus.cc:3037] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.564985 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de03fdd63bd347dea4eb6e5d410da68e" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:31.565191 7271 raft_consensus.cc:3037] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.565776 6877 raft_consensus.cc:2454] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:31.565912 7271 raft_consensus.cc:2454] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:31.566041 6945 leader_election.cc:302] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:31.566187 8201 raft_consensus.cc:2781] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.566287 8201 raft_consensus.cc:686] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:31.566640 8201 consensus_queue.cc:237] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.568341 6737 catalog_manager.cc:5484] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:31.595896 7974 raft_consensus.cc:482] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.596083 7974 raft_consensus.cc:504] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.596472 7974 leader_election.cc:288] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:31.596873 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf2f8d747d4b2694a63034b86635b6" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:31.596885 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf2f8d747d4b2694a63034b86635b6" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:31.597003 7140 raft_consensus.cc:2454] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:31.597004 6877 raft_consensus.cc:2454] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:31.597369 7206 leader_election.cc:302] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.597517 7974 raft_consensus.cc:2781] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.597575 7974 raft_consensus.cc:482] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.597611 7974 raft_consensus.cc:3037] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.598502 7974 raft_consensus.cc:504] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.598800 7974 leader_election.cc:288] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:31.599099 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf2f8d747d4b2694a63034b86635b6" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:31.599210 7140 raft_consensus.cc:3037] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.599375 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf2f8d747d4b2694a63034b86635b6" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:31.599478 6877 raft_consensus.cc:3037] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.600159 7140 raft_consensus.cc:2454] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:31.600412 6877 raft_consensus.cc:2454] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:31.600431 7206 leader_election.cc:302] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.600589 7974 raft_consensus.cc:2781] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.600833 7974 raft_consensus.cc:686] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:31.600980 7974 consensus_queue.cc:237] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:31.602627 6737 catalog_manager.cc:5484] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
W20240501 07:58:31.733379 8295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.733628 8295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.736716 8295 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 07:58:31.821499 6878 raft_consensus.cc:1261] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.821996 8290 consensus_queue.cc:1035] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.823211 7271 raft_consensus.cc:1261] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.824047 8290 consensus_queue.cc:1035] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.827463 7272 raft_consensus.cc:1261] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.827891 8201 consensus_queue.cc:1035] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.831362 6878 raft_consensus.cc:1261] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.831892 8201 consensus_queue.cc:1035] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.835685 7140 raft_consensus.cc:1261] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.835846 6878 raft_consensus.cc:1261] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:31.836230 8293 consensus_queue.cc:1035] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.836551 8293 consensus_queue.cc:1035] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:31.847203 8252 mvcc.cc:205] Tried to move back new op lower bound from 7022798077216849920 to 7022798076100898816. Current Snapshot: MvccSnapshot[applied={T|T < 7022798077216849920}]
I20240501 07:58:31.856020 8251 mvcc.cc:205] Tried to move back new op lower bound from 7022798077275353088 to 7022798076318994432. Current Snapshot: MvccSnapshot[applied={T|T < 7022798077275353088}]
I20240501 07:58:31.870440 8262 mvcc.cc:205] Tried to move back new op lower bound from 7022798077219188736 to 7022798076178247680. Current Snapshot: MvccSnapshot[applied={T|T < 7022798077219188736}]
INSERT report
rows total: 2000
time total: 104.272 ms
time per row: 0.0521362 ms
I20240501 07:58:31.889864 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:31.895318 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 4882718ff90c4d939917f4970a9acb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.895581 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 4882718ff90c4d939917f4970a9acb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.895676 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4882718ff90c4d939917f4970a9acb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.895701 6858 tablet_service.cc:1467] Processing CreateTablet for tablet b6421aafe6b34136bd8e1853cdf50786 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.895848 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4882718ff90c4d939917f4970a9acb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.895900 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6421aafe6b34136bd8e1853cdf50786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.896173 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 4882718ff90c4d939917f4970a9acb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.896430 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4882718ff90c4d939917f4970a9acb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.897881 8280 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:31.898015 8277 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:31.898563 8280 tablet_bootstrap.cc:654] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.898825 8277 tablet_bootstrap.cc:654] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.899015 8279 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:31.899050 7251 tablet_service.cc:1467] Processing CreateTablet for tablet b6421aafe6b34136bd8e1853cdf50786 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.899331 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6421aafe6b34136bd8e1853cdf50786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.899847 8279 tablet_bootstrap.cc:654] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.900108 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.900192 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.900324 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.900426 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.900727 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.900928 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c67c0f7ae4a458a81bbcb6b29c2fbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.902942 8286 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:31.903096 8279 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:31.903172 8277 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:31.903179 8279 ts_tablet_manager.cc:1367] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:31.903254 8277 ts_tablet_manager.cc:1367] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:31.903687 8277 raft_consensus.cc:348] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.903779 8286 tablet_bootstrap.cc:654] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.903823 8277 raft_consensus.cc:374] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.903856 8277 raft_consensus.cc:729] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.903798 8279 raft_consensus.cc:348] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.903905 8279 raft_consensus.cc:374] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.903935 8279 raft_consensus.cc:729] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.903978 8277 consensus_queue.cc:260] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.904054 8279 consensus_queue.cc:260] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.904276 8277 ts_tablet_manager.cc:1398] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:31.904351 8279 ts_tablet_manager.cc:1398] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.904450 8277 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:31.905016 8286 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:31.905097 8286 ts_tablet_manager.cc:1367] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:31.905421 8277 tablet_bootstrap.cc:654] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.905655 8286 raft_consensus.cc:348] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.905756 8286 raft_consensus.cc:374] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.905789 8286 raft_consensus.cc:729] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.905903 8286 consensus_queue.cc:260] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.906209 8286 ts_tablet_manager.cc:1398] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.906631 8286 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:31.907299 8279 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:31.904580 7120 tablet_service.cc:1467] Processing CreateTablet for tablet b6421aafe6b34136bd8e1853cdf50786 (DEFAULT_TABLE table=rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.908843 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6421aafe6b34136bd8e1853cdf50786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.909464 8279 tablet_bootstrap.cc:654] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.910547 8286 tablet_bootstrap.cc:654] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.910691 8279 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:31.910848 8279 ts_tablet_manager.cc:1367] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:31.911355 8279 raft_consensus.cc:348] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.911468 8279 raft_consensus.cc:374] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.911512 8279 raft_consensus.cc:729] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.911677 8279 consensus_queue.cc:260] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.911975 8280 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:31.912063 8280 ts_tablet_manager.cc:1367] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20240501 07:58:31.912099 8279 ts_tablet_manager.cc:1398] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:31.912118 8286 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:31.912190 8286 ts_tablet_manager.cc:1367] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:31.912256 8279 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:31.912384 8277 tablet_bootstrap.cc:492] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:31.912457 8277 ts_tablet_manager.cc:1367] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:31.912552 8280 raft_consensus.cc:348] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.912595 8286 raft_consensus.cc:348] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.912673 8286 raft_consensus.cc:374] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.912670 8280 raft_consensus.cc:374] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.912704 8286 raft_consensus.cc:729] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.912704 8280 raft_consensus.cc:729] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.912822 8280 consensus_queue.cc:260] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.912822 8286 consensus_queue.cc:260] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.912932 8277 raft_consensus.cc:348] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.913028 8277 raft_consensus.cc:374] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.913058 8277 raft_consensus.cc:729] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.913072 8286 ts_tablet_manager.cc:1398] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.913072 8280 ts_tablet_manager.cc:1398] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:31.913231 8280 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:31.913166 8277 consensus_queue.cc:260] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:31.913313 8279 tablet_bootstrap.cc:654] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.913501 8277 ts_tablet_manager.cc:1398] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.914088 8280 tablet_bootstrap.cc:654] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.914240 8279 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:31.914306 8279 ts_tablet_manager.cc:1367] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:31.914690 8279 raft_consensus.cc:348] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.914794 8279 raft_consensus.cc:374] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.914829 8279 raft_consensus.cc:729] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.914932 8279 consensus_queue.cc:260] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.915287 8279 ts_tablet_manager.cc:1398] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:31.917771 8280 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:31.917891 8280 ts_tablet_manager.cc:1367] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:58:31.918313 8280 raft_consensus.cc:348] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.918402 8280 raft_consensus.cc:374] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.918424 8280 raft_consensus.cc:729] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.918548 8280 consensus_queue.cc:260] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:31.918787 8280 ts_tablet_manager.cc:1398] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:31.954856 8084 raft_consensus.cc:482] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.954985 8084 raft_consensus.cc:504] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.955523 8084 leader_election.cc:288] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:31.955883 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4882718ff90c4d939917f4970a9acb9e" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:31.955883 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4882718ff90c4d939917f4970a9acb9e" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:31.956020 7140 raft_consensus.cc:2454] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:31.956020 7009 raft_consensus.cc:2454] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:31.956328 6815 leader_election.cc:302] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.956529 8084 raft_consensus.cc:2781] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.956596 8084 raft_consensus.cc:482] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.956638 8084 raft_consensus.cc:3037] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.957469 8084 raft_consensus.cc:504] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.957780 8084 leader_election.cc:288] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:31.958074 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4882718ff90c4d939917f4970a9acb9e" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:31.958185 7009 raft_consensus.cc:3037] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.958207 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4882718ff90c4d939917f4970a9acb9e" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:31.958313 7140 raft_consensus.cc:3037] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.959214 7009 raft_consensus.cc:2454] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:31.959214 7140 raft_consensus.cc:2454] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:31.959501 6813 leader_election.cc:302] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:31.959674 8084 raft_consensus.cc:2781] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.959806 8084 raft_consensus.cc:686] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:31.959929 8084 consensus_queue.cc:237] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:31.961639 6730 catalog_manager.cc:5484] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:32.031214 8201 raft_consensus.cc:482] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.031347 8201 raft_consensus.cc:504] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.031771 8201 leader_election.cc:288] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.032081 8084 raft_consensus.cc:482] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.032130 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67c0f7ae4a458a81bbcb6b29c2fbe7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:32.032130 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67c0f7ae4a458a81bbcb6b29c2fbe7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:32.032174 8084 raft_consensus.cc:504] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.032260 7140 raft_consensus.cc:2454] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.032260 7272 raft_consensus.cc:2454] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.032493 8084 leader_election.cc:288] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:32.032569 6944 leader_election.cc:302] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.032876 8201 raft_consensus.cc:2781] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.032876 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6421aafe6b34136bd8e1853cdf50786" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:32.032939 8201 raft_consensus.cc:482] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.032976 8201 raft_consensus.cc:3037] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.032985 7272 raft_consensus.cc:2454] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:32.032965 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6421aafe6b34136bd8e1853cdf50786" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:32.033072 7140 raft_consensus.cc:2454] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:32.033300 6813 leader_election.cc:302] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:32.033463 8084 raft_consensus.cc:2781] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.033517 8084 raft_consensus.cc:482] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.033551 8084 raft_consensus.cc:3037] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.034308 8201 raft_consensus.cc:504] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.034478 8084 raft_consensus.cc:504] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.034682 8201 leader_election.cc:288] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.035147 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6421aafe6b34136bd8e1853cdf50786" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:32.035240 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67c0f7ae4a458a81bbcb6b29c2fbe7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:32.035384 7270 raft_consensus.cc:3037] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.035594 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6421aafe6b34136bd8e1853cdf50786" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:32.035694 7140 raft_consensus.cc:3037] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.035780 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67c0f7ae4a458a81bbcb6b29c2fbe7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:32.035880 7141 raft_consensus.cc:3037] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.036406 7270 raft_consensus.cc:2454] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.036515 7140 raft_consensus.cc:2454] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:32.035241 7272 raft_consensus.cc:3037] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.036799 7141 raft_consensus.cc:2454] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.036794 6813 leader_election.cc:302] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.036912 8084 leader_election.cc:288] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:32.036985 8084 raft_consensus.cc:2781] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.037034 8084 raft_consensus.cc:686] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:32.037086 6944 leader_election.cc:302] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.037150 8084 consensus_queue.cc:237] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.037381 7272 raft_consensus.cc:2454] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:32.037837 8201 raft_consensus.cc:2781] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.038010 8201 raft_consensus.cc:686] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:32.038164 8201 consensus_queue.cc:237] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.039065 6730 catalog_manager.cc:5484] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:32.039868 6732 catalog_manager.cc:5484] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:32.166721 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:32.166980 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:32.170214 8350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 07:58:32.253113 7141 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.253118 7272 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.253664 8345 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.253806 8084 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.253875 7009 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.254330 8084 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.254434 7141 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.254707 7140 raft_consensus.cc:1261] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.255173 8290 consensus_queue.cc:1035] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.258543 8084 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.260080 7272 raft_consensus.cc:1261] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.264685 8201 consensus_queue.cc:1035] T 6c67c0f7ae4a458a81bbcb6b29c2fbe7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.78 ms
time per row: 0.0523901 ms
I20240501 07:58:32.319808 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:32.325155 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 08a748c1a5574f5faf7966f733ff2844 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.325484 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a748c1a5574f5faf7966f733ff2844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.325527 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 08a748c1a5574f5faf7966f733ff2844 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.325613 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 08a748c1a5574f5faf7966f733ff2844 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.325866 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a748c1a5574f5faf7966f733ff2844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.325861 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a748c1a5574f5faf7966f733ff2844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.327340 8280 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:32.328038 8279 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:32.328097 8280 tablet_bootstrap.cc:654] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.328971 8279 tablet_bootstrap.cc:654] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.328944 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 338a7d3de39641278a7268d61ba78795 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.329173 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338a7d3de39641278a7268d61ba78795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.329320 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 6f251432f70a417fa8be862182b16d91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.329361 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 6f251432f70a417fa8be862182b16d91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.329572 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f251432f70a417fa8be862182b16d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.329572 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f251432f70a417fa8be862182b16d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.329609 8277 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:32.330469 8277 tablet_bootstrap.cc:654] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.330737 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 338a7d3de39641278a7268d61ba78795 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.330967 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338a7d3de39641278a7268d61ba78795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.331534 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 6f251432f70a417fa8be862182b16d91 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.331801 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f251432f70a417fa8be862182b16d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.331807 8277 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:32.331872 8277 ts_tablet_manager.cc:1367] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.332288 8277 raft_consensus.cc:348] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.332368 8280 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:32.332381 8277 raft_consensus.cc:374] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.332410 8277 raft_consensus.cc:729] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.332455 8280 ts_tablet_manager.cc:1367] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:32.332528 8277 consensus_queue.cc:260] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.332805 8277 ts_tablet_manager.cc:1398] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.332967 8277 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:32.332968 8280 raft_consensus.cc:348] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.333088 8280 raft_consensus.cc:374] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.333123 8280 raft_consensus.cc:729] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.333269 8280 consensus_queue.cc:260] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.333596 8280 ts_tablet_manager.cc:1398] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.333789 8277 tablet_bootstrap.cc:654] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.333900 8280 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:32.334087 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 338a7d3de39641278a7268d61ba78795 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.334322 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338a7d3de39641278a7268d61ba78795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.335041 8280 tablet_bootstrap.cc:654] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.335115 8277 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:32.335192 8277 ts_tablet_manager.cc:1367] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.335601 8277 raft_consensus.cc:348] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.335700 8277 raft_consensus.cc:374] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.335728 8277 raft_consensus.cc:729] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.335847 8277 consensus_queue.cc:260] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.335924 8286 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:32.336097 8280 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:32.336112 8277 ts_tablet_manager.cc:1398] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.336182 8280 ts_tablet_manager.cc:1367] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:32.336647 8280 raft_consensus.cc:348] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.336740 8280 raft_consensus.cc:374] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.336768 8286 tablet_bootstrap.cc:654] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.336782 8280 raft_consensus.cc:729] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.336930 8280 consensus_queue.cc:260] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.337188 8280 ts_tablet_manager.cc:1398] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:32.339165 8286 tablet_bootstrap.cc:492] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:32.339275 8286 ts_tablet_manager.cc:1367] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.339699 8286 raft_consensus.cc:348] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.339830 8286 raft_consensus.cc:374] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.339886 8286 raft_consensus.cc:729] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.340019 8286 consensus_queue.cc:260] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.340333 8286 ts_tablet_manager.cc:1398] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.340551 8286 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:32.340795 8280 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:32.341449 8286 tablet_bootstrap.cc:654] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.341538 8279 tablet_bootstrap.cc:492] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:32.341630 8279 ts_tablet_manager.cc:1367] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 07:58:32.341840 8280 tablet_bootstrap.cc:654] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.342070 8279 raft_consensus.cc:348] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.342180 8279 raft_consensus.cc:374] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.342222 8279 raft_consensus.cc:729] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.342360 8279 consensus_queue.cc:260] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.343405 8286 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:32.343489 8280 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:32.343497 8286 ts_tablet_manager.cc:1367] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:32.343580 8280 ts_tablet_manager.cc:1367] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:32.343974 8286 raft_consensus.cc:348] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.344008 8280 raft_consensus.cc:348] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.344084 8286 raft_consensus.cc:374] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.344094 8280 raft_consensus.cc:374] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.344121 8286 raft_consensus.cc:729] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.344137 8280 raft_consensus.cc:729] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.344238 8280 consensus_queue.cc:260] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.344252 8286 consensus_queue.cc:260] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.344466 8280 ts_tablet_manager.cc:1398] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:32.344578 8286 ts_tablet_manager.cc:1398] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.345232 8279 ts_tablet_manager.cc:1398] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:32.345438 8279 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:32.346278 8279 tablet_bootstrap.cc:654] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.347656 8279 tablet_bootstrap.cc:492] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:32.347757 8279 ts_tablet_manager.cc:1367] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:32.348014 8386 raft_consensus.cc:482] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.348115 8386 raft_consensus.cc:504] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.348271 8279 raft_consensus.cc:348] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.348377 8279 raft_consensus.cc:374] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.348418 8279 raft_consensus.cc:729] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.348590 8386 leader_election.cc:288] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.348582 8279 consensus_queue.cc:260] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.348873 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f251432f70a417fa8be862182b16d91" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:32.349027 6878 raft_consensus.cc:2454] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.348999 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f251432f70a417fa8be862182b16d91" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:32.349123 7272 raft_consensus.cc:2454] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.349347 6945 leader_election.cc:302] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.349498 8386 raft_consensus.cc:2781] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.349574 8386 raft_consensus.cc:482] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.349627 8386 raft_consensus.cc:3037] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.349762 8279 ts_tablet_manager.cc:1398] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.350565 8386 raft_consensus.cc:504] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.350935 8386 leader_election.cc:288] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.351218 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f251432f70a417fa8be862182b16d91" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:32.351351 6878 raft_consensus.cc:3037] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.351388 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f251432f70a417fa8be862182b16d91" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:32.351503 7272 raft_consensus.cc:3037] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.352437 6878 raft_consensus.cc:2454] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.352437 7272 raft_consensus.cc:2454] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.352701 6945 leader_election.cc:302] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.353060 8290 raft_consensus.cc:2781] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.353152 8290 raft_consensus.cc:686] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:32.353282 8290 consensus_queue.cc:237] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.354974 6732 catalog_manager.cc:5484] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:32.361888 8387 raft_consensus.cc:482] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.362015 8387 raft_consensus.cc:504] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.362418 8387 leader_election.cc:288] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:32.362836 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a748c1a5574f5faf7966f733ff2844" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:32.362830 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a748c1a5574f5faf7966f733ff2844" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:32.362968 7009 raft_consensus.cc:2454] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:32.362968 7141 raft_consensus.cc:2454] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:32.363308 6815 leader_election.cc:302] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.363474 8387 raft_consensus.cc:2781] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.363533 8387 raft_consensus.cc:482] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.363564 8387 raft_consensus.cc:3037] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.366655 8387 raft_consensus.cc:504] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.367412 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a748c1a5574f5faf7966f733ff2844" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:32.367549 7141 raft_consensus.cc:3037] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.368412 7141 raft_consensus.cc:2454] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:32.368675 6813 leader_election.cc:302] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.368835 8084 raft_consensus.cc:2781] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.368876 8387 leader_election.cc:288] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:32.368911 8084 raft_consensus.cc:686] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:32.369030 8084 consensus_queue.cc:237] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:32.370107 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a748c1a5574f5faf7966f733ff2844" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:32.370225 7009 raft_consensus.cc:3037] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.371147 7009 raft_consensus.cc:2454] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:32.371881 6732 catalog_manager.cc:5484] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:32.406940 8349 raft_consensus.cc:482] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.407091 8349 raft_consensus.cc:504] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.407472 8349 leader_election.cc:288] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:32.407804 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338a7d3de39641278a7268d61ba78795" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:32.407804 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338a7d3de39641278a7268d61ba78795" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:32.407943 7272 raft_consensus.cc:2454] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:32.407943 6878 raft_consensus.cc:2454] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:32.408257 7075 leader_election.cc:302] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.408475 8349 raft_consensus.cc:2781] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.408548 8349 raft_consensus.cc:482] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.408579 8349 raft_consensus.cc:3037] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.409541 8349 raft_consensus.cc:504] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.409879 8349 leader_election.cc:288] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:32.410130 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338a7d3de39641278a7268d61ba78795" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:32.410245 7272 raft_consensus.cc:3037] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.410252 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338a7d3de39641278a7268d61ba78795" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:32.410399 6878 raft_consensus.cc:3037] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.411224 7272 raft_consensus.cc:2454] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:32.411335 6878 raft_consensus.cc:2454] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:32.411578 7075 leader_election.cc:302] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:32.411782 8349 raft_consensus.cc:2781] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.411867 8349 raft_consensus.cc:686] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:32.412000 8349 consensus_queue.cc:237] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:32.413692 6732 catalog_manager.cc:5484] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:32.541555 8402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:32.541803 8402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:32.544831 8402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 07:58:32.634054 7009 raft_consensus.cc:1261] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.634512 8345 consensus_queue.cc:1035] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.635782 6878 raft_consensus.cc:1261] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.636518 7270 raft_consensus.cc:1261] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.636901 8349 consensus_queue.cc:1035] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.637655 8258 consensus_queue.cc:1035] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.639521 7141 raft_consensus.cc:1261] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.639664 7272 raft_consensus.cc:1261] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.644134 6879 raft_consensus.cc:1261] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.645928 8386 consensus_queue.cc:1035] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.646080 8290 consensus_queue.cc:1035] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.646309 8345 consensus_queue.cc:1035] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.444 ms
time per row: 0.0547222 ms
I20240501 07:58:32.699239 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:32.705005 7251 tablet_service.cc:1467] Processing CreateTablet for tablet c2f299abad9e415283470b4b214fc62f (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.705116 6988 tablet_service.cc:1467] Processing CreateTablet for tablet c2f299abad9e415283470b4b214fc62f (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.705376 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f299abad9e415283470b4b214fc62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.705425 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f299abad9e415283470b4b214fc62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.705902 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 651beb63e03d40bca88f01ebed416754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.705962 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 69ff85c89fa443539b28c0d0cc299cfd (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.706135 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651beb63e03d40bca88f01ebed416754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.706337 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 651beb63e03d40bca88f01ebed416754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.706593 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651beb63e03d40bca88f01ebed416754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.706539 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 651beb63e03d40bca88f01ebed416754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.706756 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651beb63e03d40bca88f01ebed416754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.707867 8286 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:32.708626 8279 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:32.708626 8277 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:32.708900 8286 tablet_bootstrap.cc:654] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.709425 8279 tablet_bootstrap.cc:654] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.709425 8277 tablet_bootstrap.cc:654] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.709803 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 69ff85c89fa443539b28c0d0cc299cfd (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.710036 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ff85c89fa443539b28c0d0cc299cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.710327 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ff85c89fa443539b28c0d0cc299cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.710340 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c2f299abad9e415283470b4b214fc62f (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.710553 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f299abad9e415283470b4b214fc62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.711580 8277 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:32.711661 8277 ts_tablet_manager.cc:1367] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:32.712157 8277 raft_consensus.cc:348] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.712276 8277 raft_consensus.cc:374] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.712309 8277 raft_consensus.cc:729] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.712364 8280 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:32.712435 8277 consensus_queue.cc:260] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.712630 8286 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:32.712702 8286 ts_tablet_manager.cc:1367] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:32.712733 8277 ts_tablet_manager.cc:1398] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.712889 8277 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:32.713145 8286 raft_consensus.cc:348] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.713238 8280 tablet_bootstrap.cc:654] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.713265 8286 raft_consensus.cc:374] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.713299 8286 raft_consensus.cc:729] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.713418 8286 consensus_queue.cc:260] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.713714 8286 ts_tablet_manager.cc:1398] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.713732 8277 tablet_bootstrap.cc:654] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.715930 8277 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:32.716012 8277 ts_tablet_manager.cc:1367] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:32.716110 8280 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:32.716110 8279 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:32.716192 8280 ts_tablet_manager.cc:1367] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:32.716192 8279 ts_tablet_manager.cc:1367] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:32.716424 8277 raft_consensus.cc:348] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.716531 8277 raft_consensus.cc:374] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.716560 8277 raft_consensus.cc:729] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.716621 8280 raft_consensus.cc:348] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.716621 8279 raft_consensus.cc:348] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.716719 8279 raft_consensus.cc:374] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.716719 8280 raft_consensus.cc:374] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.716688 8277 consensus_queue.cc:260] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.716753 8280 raft_consensus.cc:729] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.716753 8279 raft_consensus.cc:729] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.716863 8279 consensus_queue.cc:260] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.716863 8280 consensus_queue.cc:260] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.716964 8277 ts_tablet_manager.cc:1398] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.717113 8280 ts_tablet_manager.cc:1398] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:32.717128 8279 ts_tablet_manager.cc:1398] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.717139 8277 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:32.717278 8279 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:32.718135 8279 tablet_bootstrap.cc:654] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.719159 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 69ff85c89fa443539b28c0d0cc299cfd (DEFAULT_TABLE table=rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.719393 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ff85c89fa443539b28c0d0cc299cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.719722 8277 tablet_bootstrap.cc:654] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.721701 8279 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:32.721788 8279 ts_tablet_manager.cc:1367] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:32.722043 8277 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:32.722108 8277 ts_tablet_manager.cc:1367] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:58:32.722219 8279 raft_consensus.cc:348] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.722309 8279 raft_consensus.cc:374] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.722338 8279 raft_consensus.cc:729] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.722451 8279 consensus_queue.cc:260] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.722560 8277 raft_consensus.cc:348] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.722643 8277 raft_consensus.cc:374] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.722672 8277 raft_consensus.cc:729] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.722733 8279 ts_tablet_manager.cc:1398] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.722791 8277 consensus_queue.cc:260] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.722868 8279 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:32.723078 8277 ts_tablet_manager.cc:1398] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:32.723606 8286 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:32.724018 8279 tablet_bootstrap.cc:654] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.725257 8279 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:32.725328 8279 ts_tablet_manager.cc:1367] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.725487 8286 tablet_bootstrap.cc:654] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.725625 8279 raft_consensus.cc:348] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.725739 8279 raft_consensus.cc:374] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.725764 8279 raft_consensus.cc:729] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.725862 8279 consensus_queue.cc:260] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.726099 8279 ts_tablet_manager.cc:1398] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.726940 8286 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:32.727051 8286 ts_tablet_manager.cc:1367] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.727422 8286 raft_consensus.cc:348] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.727555 8286 raft_consensus.cc:374] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.727593 8286 raft_consensus.cc:729] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.727705 8286 consensus_queue.cc:260] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.728047 8286 ts_tablet_manager.cc:1398] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.867172 8386 raft_consensus.cc:482] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.867334 8386 raft_consensus.cc:504] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.867823 8386 leader_election.cc:288] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:32.868294 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:32.868312 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:32.868443 7141 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.868472 6879 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.868906 6945 leader_election.cc:302] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.869057 8386 raft_consensus.cc:2781] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.869109 8386 raft_consensus.cc:482] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.869153 8386 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.870214 8386 raft_consensus.cc:504] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.870543 8386 leader_election.cc:288] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:32.870785 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:32.870903 6878 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.870899 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:32.871063 7141 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.871878 6878 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.872004 7141 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.872169 6945 leader_election.cc:302] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.872344 8386 raft_consensus.cc:2781] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.872416 8386 raft_consensus.cc:686] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:32.872529 8386 consensus_queue.cc:237] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:32.874092 6732 catalog_manager.cc:5484] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:32.876430 8386 raft_consensus.cc:482] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.876539 8386 raft_consensus.cc:504] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.876896 8386 leader_election.cc:288] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.877270 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651beb63e03d40bca88f01ebed416754" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:32.877346 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651beb63e03d40bca88f01ebed416754" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:32.877414 7270 raft_consensus.cc:2454] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.877475 7141 raft_consensus.cc:2454] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:32.877717 6944 leader_election.cc:302] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.877871 8386 raft_consensus.cc:2781] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.877925 8386 raft_consensus.cc:482] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.878007 8386 raft_consensus.cc:3037] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.878684 8386 raft_consensus.cc:504] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.878973 8386 leader_election.cc:288] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:32.879267 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651beb63e03d40bca88f01ebed416754" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:32.879324 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651beb63e03d40bca88f01ebed416754" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:32.879376 7141 raft_consensus.cc:3037] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.879434 7270 raft_consensus.cc:3037] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.880316 7141 raft_consensus.cc:2454] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.880317 7270 raft_consensus.cc:2454] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:32.880609 6944 leader_election.cc:302] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:32.880806 8386 raft_consensus.cc:2781] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.880903 8386 raft_consensus.cc:686] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:32.881031 8386 consensus_queue.cc:237] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:32.882684 6732 catalog_manager.cc:5484] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:33.055271 8290 raft_consensus.cc:482] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.055404 8290 raft_consensus.cc:504] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.055801 8290 leader_election.cc:288] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:33.056141 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f299abad9e415283470b4b214fc62f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:33.056195 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f299abad9e415283470b4b214fc62f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:33.056299 7270 raft_consensus.cc:2454] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:33.056309 7141 raft_consensus.cc:2454] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:33.056598 6944 leader_election.cc:302] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:33.056772 8290 raft_consensus.cc:2781] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.056828 8290 raft_consensus.cc:482] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.056854 8290 raft_consensus.cc:3037] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.057716 8290 raft_consensus.cc:504] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.058037 8290 leader_election.cc:288] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:33.058382 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f299abad9e415283470b4b214fc62f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:33.058375 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f299abad9e415283470b4b214fc62f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:33.058483 7141 raft_consensus.cc:3037] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.058487 7270 raft_consensus.cc:3037] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.059242 7141 raft_consensus.cc:2454] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:33.059365 7270 raft_consensus.cc:2454] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:33.059512 6944 leader_election.cc:302] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:33.059708 8290 raft_consensus.cc:2781] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.059836 8290 raft_consensus.cc:686] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:33.059971 8290 consensus_queue.cc:237] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.061676 6732 catalog_manager.cc:5484] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
W20240501 07:58:33.189092 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.189366 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.192790 8453 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 07:58:33.276675 7272 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.276906 7141 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.277267 8290 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.277408 8386 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.279260 7270 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.279922 7139 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.280161 7140 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.280568 8386 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.280822 8386 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.281242 6878 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.281867 8483 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.283100 8483 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.591 ms
time per row: 0.0547953 ms
I20240501 07:58:33.345798 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:33.351233 6857 tablet_service.cc:1467] Processing CreateTablet for tablet a42d4784867744b6be9e343ec6fdedbc (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.351317 7250 tablet_service.cc:1467] Processing CreateTablet for tablet a42d4784867744b6be9e343ec6fdedbc (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.351315 6988 tablet_service.cc:1467] Processing CreateTablet for tablet a42d4784867744b6be9e343ec6fdedbc (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.351570 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42d4784867744b6be9e343ec6fdedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.351570 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42d4784867744b6be9e343ec6fdedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.351637 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42d4784867744b6be9e343ec6fdedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.353772 8513 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:33.353889 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 37e73deb03274bba82ceb67087208bdd (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.354130 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e73deb03274bba82ceb67087208bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.354646 8513 tablet_bootstrap.cc:654] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.354810 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.355062 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b3a231d67e4092a94d3539e31e62f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.355156 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 37e73deb03274bba82ceb67087208bdd (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.355403 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e73deb03274bba82ceb67087208bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.355459 8512 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:33.356287 8512 tablet_bootstrap.cc:654] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.356287 6990 tablet_service.cc:1467] Processing CreateTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.356523 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b3a231d67e4092a94d3539e31e62f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.357626 8516 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:33.357720 8512 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:33.357802 8512 ts_tablet_manager.cc:1367] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:33.358189 8512 raft_consensus.cc:348] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.358310 8512 raft_consensus.cc:374] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.358343 8512 raft_consensus.cc:729] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.358490 8512 consensus_queue.cc:260] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.358575 8516 tablet_bootstrap.cc:654] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.358872 8512 ts_tablet_manager.cc:1398] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.359112 8512 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:33.359306 6858 tablet_service.cc:1467] Processing CreateTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.359535 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b3a231d67e4092a94d3539e31e62f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.359719 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 37e73deb03274bba82ceb67087208bdd (DEFAULT_TABLE table=rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.359807 8516 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:33.359872 8516 ts_tablet_manager.cc:1367] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:33.359961 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e73deb03274bba82ceb67087208bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.360298 8516 raft_consensus.cc:348] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.360386 8516 raft_consensus.cc:374] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.360414 8516 raft_consensus.cc:729] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.360528 8516 consensus_queue.cc:260] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.360807 8516 ts_tablet_manager.cc:1398] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.361160 8512 tablet_bootstrap.cc:654] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.363698 8513 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:33.363785 8513 ts_tablet_manager.cc:1367] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 07:58:33.364187 8513 raft_consensus.cc:348] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.364277 8513 raft_consensus.cc:374] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.364324 8513 raft_consensus.cc:729] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.364454 8513 consensus_queue.cc:260] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.364718 8513 ts_tablet_manager.cc:1398] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:33.364877 8513 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:33.365713 8513 tablet_bootstrap.cc:654] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.367219 8511 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:33.367825 8513 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:33.367942 8513 ts_tablet_manager.cc:1367] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:33.368070 8511 tablet_bootstrap.cc:654] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.368363 8513 raft_consensus.cc:348] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.368477 8513 raft_consensus.cc:374] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.368525 8513 raft_consensus.cc:729] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.368706 8513 consensus_queue.cc:260] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.369001 8513 ts_tablet_manager.cc:1398] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:33.369174 8513 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:33.369185 8512 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:33.369266 8512 ts_tablet_manager.cc:1367] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:58:33.369693 8512 raft_consensus.cc:348] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.369786 8512 raft_consensus.cc:374] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.369817 8512 raft_consensus.cc:729] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.369939 8512 consensus_queue.cc:260] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.370008 8513 tablet_bootstrap.cc:654] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.370242 8512 ts_tablet_manager.cc:1398] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:33.370407 8512 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:33.371181 8512 tablet_bootstrap.cc:654] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.371618 8511 tablet_bootstrap.cc:492] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:33.371685 8511 ts_tablet_manager.cc:1367] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:33.372071 8511 raft_consensus.cc:348] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.372164 8511 raft_consensus.cc:374] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.372195 8511 raft_consensus.cc:729] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.372370 8511 consensus_queue.cc:260] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.372702 8511 ts_tablet_manager.cc:1398] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.372822 8511 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:33.373392 8513 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:33.373458 8513 ts_tablet_manager.cc:1367] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:33.373579 8511 tablet_bootstrap.cc:654] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.373863 8513 raft_consensus.cc:348] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.373947 8513 raft_consensus.cc:374] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.373978 8513 raft_consensus.cc:729] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.374091 8513 consensus_queue.cc:260] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.374372 8513 ts_tablet_manager.cc:1398] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:33.375471 8511 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:33.375591 8511 ts_tablet_manager.cc:1367] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.375982 8511 raft_consensus.cc:348] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.376200 8511 raft_consensus.cc:374] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.376255 8511 raft_consensus.cc:729] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.376377 8511 consensus_queue.cc:260] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.376672 8511 ts_tablet_manager.cc:1398] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.378079 8512 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:33.378158 8512 ts_tablet_manager.cc:1367] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:33.378540 8512 raft_consensus.cc:348] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.378633 8512 raft_consensus.cc:374] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.378671 8512 raft_consensus.cc:729] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.378793 8512 consensus_queue.cc:260] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.379060 8512 ts_tablet_manager.cc:1398] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.416314 8258 raft_consensus.cc:482] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.416445 8258 raft_consensus.cc:504] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.416806 8258 leader_election.cc:288] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:33.417174 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e73deb03274bba82ceb67087208bdd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:33.417174 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e73deb03274bba82ceb67087208bdd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:33.417325 6878 raft_consensus.cc:2454] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:33.417327 7270 raft_consensus.cc:2454] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:33.417606 7075 leader_election.cc:302] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:33.417760 8258 raft_consensus.cc:2781] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.417827 8258 raft_consensus.cc:482] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.417874 8258 raft_consensus.cc:3037] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.418798 8258 raft_consensus.cc:504] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.419124 8258 leader_election.cc:288] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:33.419432 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e73deb03274bba82ceb67087208bdd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:33.419564 6878 raft_consensus.cc:3037] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.419611 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e73deb03274bba82ceb67087208bdd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:33.419720 7270 raft_consensus.cc:3037] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.420547 6878 raft_consensus.cc:2454] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:33.420653 7270 raft_consensus.cc:2454] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:33.420850 7076 leader_election.cc:302] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:33.421209 8258 raft_consensus.cc:2781] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.421305 8258 raft_consensus.cc:686] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:33.421445 8258 consensus_queue.cc:237] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:33.423285 6733 catalog_manager.cc:5484] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:33.565755 8293 raft_consensus.cc:482] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.565871 8293 raft_consensus.cc:504] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.566270 8293 leader_election.cc:288] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:33.566612 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b3a231d67e4092a94d3539e31e62f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:33.566634 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b3a231d67e4092a94d3539e31e62f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:33.566736 6878 raft_consensus.cc:2454] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:33.566766 7010 raft_consensus.cc:2454] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:33.567041 7207 leader_election.cc:302] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:33.567209 8293 raft_consensus.cc:2781] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.567265 8293 raft_consensus.cc:482] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.567291 8293 raft_consensus.cc:3037] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.568228 8293 raft_consensus.cc:504] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.568521 8293 leader_election.cc:288] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:33.568861 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b3a231d67e4092a94d3539e31e62f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:33.568972 6878 raft_consensus.cc:3037] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.568939 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b3a231d67e4092a94d3539e31e62f4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:33.569065 7010 raft_consensus.cc:3037] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.569813 6878 raft_consensus.cc:2454] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:33.569844 7010 raft_consensus.cc:2454] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:33.570115 7208 leader_election.cc:302] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:33.570281 8293 raft_consensus.cc:2781] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.570468 8293 raft_consensus.cc:686] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:33.570621 8293 consensus_queue.cc:237] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:33.572511 6733 catalog_manager.cc:5484] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:33.667171 8293 raft_consensus.cc:482] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.667291 8293 raft_consensus.cc:504] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.667666 8293 leader_election.cc:288] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:33.668076 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42d4784867744b6be9e343ec6fdedbc" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:33.668062 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42d4784867744b6be9e343ec6fdedbc" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:33.668227 7010 raft_consensus.cc:2454] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:33.668227 6878 raft_consensus.cc:2454] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:33.668514 7207 leader_election.cc:302] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:33.668677 8293 raft_consensus.cc:2781] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.668740 8293 raft_consensus.cc:482] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.668768 8293 raft_consensus.cc:3037] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.669683 8293 raft_consensus.cc:504] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.670006 8293 leader_election.cc:288] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:33.670364 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42d4784867744b6be9e343ec6fdedbc" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:33.670456 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42d4784867744b6be9e343ec6fdedbc" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:33.670532 6878 raft_consensus.cc:3037] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.670558 7010 raft_consensus.cc:3037] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.671520 6878 raft_consensus.cc:2454] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:33.671519 7010 raft_consensus.cc:2454] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:33.671790 7207 leader_election.cc:302] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:33.671969 8293 raft_consensus.cc:2781] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.672037 8293 raft_consensus.cc:686] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:33.672178 8293 consensus_queue.cc:237] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.674516 6733 catalog_manager.cc:5484] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:33.829146 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.829370 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.832427 8528 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 07:58:33.917330 6878 raft_consensus.cc:1261] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.917603 7010 raft_consensus.cc:1261] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.917812 8293 consensus_queue.cc:1035] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.918025 8525 consensus_queue.cc:1035] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.920011 6879 raft_consensus.cc:1261] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.920374 7010 raft_consensus.cc:1261] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.920748 8525 consensus_queue.cc:1035] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.921283 6879 raft_consensus.cc:1261] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.921842 7270 raft_consensus.cc:1261] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.921970 8349 consensus_queue.cc:1035] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.922349 8525 consensus_queue.cc:1035] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.923769 8349 consensus_queue.cc:1035] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.940812 8567 mvcc.cc:205] Tried to move back new op lower bound from 7022798085801734144 to 7022798084386758656. Current Snapshot: MvccSnapshot[applied={T|T < 7022798085801734144}]
INSERT report
rows total: 2000
time total: 108.332 ms
time per row: 0.0541662 ms
I20240501 07:58:33.987684 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:33.992930 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 27f537b776474b2eb6f8065a29a1e344 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.993131 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 27f537b776474b2eb6f8065a29a1e344 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.993260 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f537b776474b2eb6f8065a29a1e344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.993423 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f537b776474b2eb6f8065a29a1e344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.993933 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 27f537b776474b2eb6f8065a29a1e344 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.994190 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f537b776474b2eb6f8065a29a1e344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.995286 6988 tablet_service.cc:1467] Processing CreateTablet for tablet c80723cb0ce4446483a5f1754e066081 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.995551 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80723cb0ce4446483a5f1754e066081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.995656 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f613e16c245de83fde8c277b8d350 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.995862 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f613e16c245de83fde8c277b8d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.996666 8590 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:33.997622 8591 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:33.997941 8590 tablet_bootstrap.cc:654] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.998485 8591 tablet_bootstrap.cc:654] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.998718 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c80723cb0ce4446483a5f1754e066081 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.999091 8590 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:33.999209 8590 ts_tablet_manager.cc:1367] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.998718 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f613e16c245de83fde8c277b8d350 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.999321 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f613e16c245de83fde8c277b8d350 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.999575 8589 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:33.999585 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f613e16c245de83fde8c277b8d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.999634 8590 raft_consensus.cc:348] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:33.999769 8590 raft_consensus.cc:374] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.999802 8590 raft_consensus.cc:729] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.999927 8590 consensus_queue.cc:260] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.000193 8590 ts_tablet_manager.cc:1398] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:34.000356 8590 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:34.000432 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80723cb0ce4446483a5f1754e066081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.001075 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f613e16c245de83fde8c277b8d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.001093 8590 tablet_bootstrap.cc:654] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.001684 8589 tablet_bootstrap.cc:654] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.999322 7250 tablet_service.cc:1467] Processing CreateTablet for tablet c80723cb0ce4446483a5f1754e066081 (DEFAULT_TABLE table=rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.001987 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80723cb0ce4446483a5f1754e066081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.002169 8590 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:34.002249 8590 ts_tablet_manager.cc:1367] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.002615 8590 raft_consensus.cc:348] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.002705 8590 raft_consensus.cc:374] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.002743 8590 raft_consensus.cc:729] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.002856 8590 consensus_queue.cc:260] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.003257 8590 ts_tablet_manager.cc:1398] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.003437 8597 raft_consensus.cc:482] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.003499 8597 raft_consensus.cc:504] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.004106 8595 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:34.004312 8597 leader_election.cc:288] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:34.004470 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80723cb0ce4446483a5f1754e066081" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
W20240501 07:58:34.004570 6944 leader_election.cc:341] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Not found: Tablet not found: c80723cb0ce4446483a5f1754e066081
W20240501 07:58:34.004845 6944 leader_election.cc:341] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:34.004914 6944 leader_election.cc:302] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2; no voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be
I20240501 07:58:34.005414 8595 tablet_bootstrap.cc:654] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.005659 8597 raft_consensus.cc:2726] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:34.006433 8591 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:34.006521 8591 ts_tablet_manager.cc:1367] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:58:34.006959 8591 raft_consensus.cc:348] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.007120 8591 raft_consensus.cc:374] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.007177 8591 raft_consensus.cc:729] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.007221 8589 tablet_bootstrap.cc:492] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:34.007318 8589 ts_tablet_manager.cc:1367] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:34.007319 8591 consensus_queue.cc:260] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.007607 8591 ts_tablet_manager.cc:1398] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.007800 8591 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:34.007762 8589 raft_consensus.cc:348] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.007853 8589 raft_consensus.cc:374] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.007889 8589 raft_consensus.cc:729] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.008023 8589 consensus_queue.cc:260] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.008376 8589 ts_tablet_manager.cc:1398] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.008528 8589 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:34.008677 8591 tablet_bootstrap.cc:654] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.009375 8589 tablet_bootstrap.cc:654] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.010406 8589 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:34.010465 8589 ts_tablet_manager.cc:1367] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.010746 8589 raft_consensus.cc:348] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.010811 8589 raft_consensus.cc:374] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.010830 8589 raft_consensus.cc:729] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.010919 8589 consensus_queue.cc:260] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.011162 8589 ts_tablet_manager.cc:1398] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.011276 8589 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:34.008080 8595 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:34.011912 8589 tablet_bootstrap.cc:654] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.012413 8595 ts_tablet_manager.cc:1367] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:34.012791 8591 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:34.012869 8591 ts_tablet_manager.cc:1367] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:34.013228 8589 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:34.013294 8589 ts_tablet_manager.cc:1367] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.013312 8591 raft_consensus.cc:348] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.013404 8591 raft_consensus.cc:374] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.013435 8591 raft_consensus.cc:729] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.013566 8591 consensus_queue.cc:260] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.013685 8589 raft_consensus.cc:348] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.013759 8589 raft_consensus.cc:374] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.013787 8589 raft_consensus.cc:729] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.013773 8595 raft_consensus.cc:348] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.013846 8591 ts_tablet_manager.cc:1398] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:34.013883 8595 raft_consensus.cc:374] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.013917 8595 raft_consensus.cc:729] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.013890 8589 consensus_queue.cc:260] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.014055 8595 consensus_queue.cc:260] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.014185 8589 ts_tablet_manager.cc:1398] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.014611 8595 ts_tablet_manager.cc:1398] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.014791 8595 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:34.015870 8595 tablet_bootstrap.cc:654] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.017452 8595 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:34.017539 8595 ts_tablet_manager.cc:1367] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.017868 8595 raft_consensus.cc:348] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.017967 8595 raft_consensus.cc:374] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.018007 8595 raft_consensus.cc:729] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.018146 8595 consensus_queue.cc:260] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.018590 8595 ts_tablet_manager.cc:1398] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.113147 8597 raft_consensus.cc:482] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.113279 8597 raft_consensus.cc:504] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.113690 8597 leader_election.cc:288] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.114100 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f537b776474b2eb6f8065a29a1e344" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:34.114159 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f537b776474b2eb6f8065a29a1e344" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:34.114225 7141 raft_consensus.cc:2454] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.114357 6878 raft_consensus.cc:2454] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.114542 6944 leader_election.cc:302] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.114730 8597 raft_consensus.cc:2781] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.114801 8597 raft_consensus.cc:482] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.114833 8597 raft_consensus.cc:3037] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.115725 8597 raft_consensus.cc:504] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.116082 8597 leader_election.cc:288] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.116452 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f537b776474b2eb6f8065a29a1e344" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:34.116498 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f537b776474b2eb6f8065a29a1e344" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:34.116577 6878 raft_consensus.cc:3037] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.116603 7141 raft_consensus.cc:3037] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.117518 6878 raft_consensus.cc:2454] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.117554 7141 raft_consensus.cc:2454] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.117830 6944 leader_election.cc:302] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.117976 8597 raft_consensus.cc:2781] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.118065 8597 raft_consensus.cc:686] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:34.118220 8597 consensus_queue.cc:237] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.120070 6733 catalog_manager.cc:5484] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:34.257508 8349 raft_consensus.cc:482] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.257634 8349 raft_consensus.cc:504] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.257987 8349 leader_election.cc:288] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:34.258385 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80723cb0ce4446483a5f1754e066081" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:34.258424 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80723cb0ce4446483a5f1754e066081" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:34.258540 7270 raft_consensus.cc:2454] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:34.258548 7010 raft_consensus.cc:2454] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:34.258832 7075 leader_election.cc:302] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.259024 8349 raft_consensus.cc:2781] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.259104 8349 raft_consensus.cc:482] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.259141 8349 raft_consensus.cc:3037] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.259972 8349 raft_consensus.cc:504] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.260282 8349 leader_election.cc:288] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:34.260603 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80723cb0ce4446483a5f1754e066081" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:34.260603 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80723cb0ce4446483a5f1754e066081" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:34.260718 7010 raft_consensus.cc:3037] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.260718 7270 raft_consensus.cc:3037] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.261636 7270 raft_consensus.cc:2454] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:34.261636 7010 raft_consensus.cc:2454] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:34.261886 7077 leader_election.cc:302] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.262048 8349 raft_consensus.cc:2781] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.262138 8349 raft_consensus.cc:686] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:34.262281 8349 consensus_queue.cc:237] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.264134 6733 catalog_manager.cc:5484] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:34.326040 8562 raft_consensus.cc:482] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.326157 8562 raft_consensus.cc:504] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.326493 8562 leader_election.cc:288] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.326798 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f613e16c245de83fde8c277b8d350" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:34.326844 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f613e16c245de83fde8c277b8d350" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:34.326961 6878 raft_consensus.cc:2454] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:34.326949 7141 raft_consensus.cc:2454] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:34.327278 7206 leader_election.cc:302] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.327450 8562 raft_consensus.cc:2781] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.327513 8562 raft_consensus.cc:482] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.327556 8562 raft_consensus.cc:3037] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.328418 8562 raft_consensus.cc:504] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.328713 8562 leader_election.cc:288] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.328966 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f613e16c245de83fde8c277b8d350" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:34.329069 7141 raft_consensus.cc:3037] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.329191 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f613e16c245de83fde8c277b8d350" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:34.329295 6878 raft_consensus.cc:3037] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.329907 7141 raft_consensus.cc:2454] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:34.329998 6878 raft_consensus.cc:2454] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:34.330178 7206 leader_election.cc:302] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.330327 8562 raft_consensus.cc:2781] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.330433 8562 raft_consensus.cc:686] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:34.330559 8562 consensus_queue.cc:237] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.332326 6733 catalog_manager.cc:5484] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:34.472764 8606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.473001 8606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.476353 8606 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 07:58:34.571355 6878 raft_consensus.cc:1261] T 27f537b776474b2eb6f8065a29a1e344 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.571516 7139 raft_consensus.cc:1261] T 27f537b776474b2eb6f8065a29a1e344 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.571811 8597 consensus_queue.cc:1035] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.571974 8483 consensus_queue.cc:1035] T 27f537b776474b2eb6f8065a29a1e344 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.575649 7010 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.575999 8349 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.576210 7270 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.576666 8349 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.590379 6878 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.590852 8525 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.591498 7139 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.591921 8562 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.595995 8642 mvcc.cc:205] Tried to move back new op lower bound from 7022798088546320384 to 7022798087219560448. Current Snapshot: MvccSnapshot[applied={T|T < 7022798088546320384}]
I20240501 07:58:34.600943 8651 mvcc.cc:205] Tried to move back new op lower bound from 7022798088498028544 to 7022798087219560448. Current Snapshot: MvccSnapshot[applied={T|T < 7022798088498028544}]
I20240501 07:58:34.602603 8646 mvcc.cc:205] Tried to move back new op lower bound from 7022798088480530432 to 7022798086629490688. Current Snapshot: MvccSnapshot[applied={T|T < 7022798088480530432}]
INSERT report
rows total: 2000
time total: 123.277 ms
time per row: 0.0616387 ms
I20240501 07:58:34.641510 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:34.646729 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 14165b2b100b4f818ccececc54bd8bad (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.646867 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 14165b2b100b4f818ccececc54bd8bad (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.647161 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14165b2b100b4f818ccececc54bd8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.647207 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14165b2b100b4f818ccececc54bd8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.647469 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 14165b2b100b4f818ccececc54bd8bad (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.647779 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14165b2b100b4f818ccececc54bd8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.649428 8673 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:34.649627 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 33e711b7a9684137969ce2bbffae40c2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.649698 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.649835 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e711b7a9684137969ce2bbffae40c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.649899 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00569cfef8704ea58ddfeb7bc7c0e20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.651052 8673 tablet_bootstrap.cc:654] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.651870 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.651928 8672 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:34.652097 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00569cfef8704ea58ddfeb7bc7c0e20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.652422 8675 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:34.652839 8672 tablet_bootstrap.cc:654] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.653414 8675 tablet_bootstrap.cc:654] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.653964 8673 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:34.654057 8673 ts_tablet_manager.cc:1367] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:58:34.654201 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 33e711b7a9684137969ce2bbffae40c2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.654428 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e711b7a9684137969ce2bbffae40c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.654593 8672 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:34.654547 8673 raft_consensus.cc:348] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.654652 8673 raft_consensus.cc:374] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.654673 8672 ts_tablet_manager.cc:1367] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.654685 8673 raft_consensus.cc:729] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.654846 8673 consensus_queue.cc:260] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.655157 8672 raft_consensus.cc:348] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.655269 8672 raft_consensus.cc:374] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.655299 8672 raft_consensus.cc:729] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.655452 8672 consensus_queue.cc:260] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.655792 8673 ts_tablet_manager.cc:1398] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:34.656292 8672 ts_tablet_manager.cc:1398] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.656455 8672 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:34.656662 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.656919 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00569cfef8704ea58ddfeb7bc7c0e20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.657397 8672 tablet_bootstrap.cc:654] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.658620 8675 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:34.658716 8675 ts_tablet_manager.cc:1367] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:34.659281 8675 raft_consensus.cc:348] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.659492 8675 raft_consensus.cc:374] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.659551 8675 raft_consensus.cc:729] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.659703 8675 consensus_queue.cc:260] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.659816 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 33e711b7a9684137969ce2bbffae40c2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.660037 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e711b7a9684137969ce2bbffae40c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.660107 8674 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:34.660552 8675 ts_tablet_manager.cc:1398] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.660712 8675 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:34.661052 8674 tablet_bootstrap.cc:654] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.662206 8674 tablet_bootstrap.cc:492] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:34.662380 8675 tablet_bootstrap.cc:654] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.662408 8674 ts_tablet_manager.cc:1367] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.662402 8673 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:34.662830 8674 raft_consensus.cc:348] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.662964 8674 raft_consensus.cc:374] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.663025 8674 raft_consensus.cc:729] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.663173 8674 consensus_queue.cc:260] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.663463 8672 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:34.663545 8672 ts_tablet_manager.cc:1367] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:34.663568 8674 ts_tablet_manager.cc:1398] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.663786 8674 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:34.663926 8672 raft_consensus.cc:348] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.664045 8672 raft_consensus.cc:374] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.664088 8672 raft_consensus.cc:729] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.664243 8672 consensus_queue.cc:260] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.664587 8673 tablet_bootstrap.cc:654] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.664659 8674 tablet_bootstrap.cc:654] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.664716 8672 ts_tablet_manager.cc:1398] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.664891 8672 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:34.665717 8674 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:34.665787 8673 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:34.665802 8674 ts_tablet_manager.cc:1367] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.665866 8673 ts_tablet_manager.cc:1367] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:34.665881 8672 tablet_bootstrap.cc:654] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.666241 8674 raft_consensus.cc:348] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.666338 8674 raft_consensus.cc:374] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.666292 8673 raft_consensus.cc:348] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.666391 8674 raft_consensus.cc:729] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.666424 8673 raft_consensus.cc:374] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.666466 8673 raft_consensus.cc:729] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.666508 8674 consensus_queue.cc:260] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.666594 8673 consensus_queue.cc:260] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.666764 8674 ts_tablet_manager.cc:1398] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.666896 8673 ts_tablet_manager.cc:1398] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.667078 8672 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:34.667147 8672 ts_tablet_manager.cc:1367] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.667543 8672 raft_consensus.cc:348] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.667639 8672 raft_consensus.cc:374] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.667673 8672 raft_consensus.cc:729] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.667778 8672 consensus_queue.cc:260] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.668025 8672 ts_tablet_manager.cc:1398] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.668814 8675 tablet_bootstrap.cc:492] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:34.668903 8675 ts_tablet_manager.cc:1367] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:34.669323 8675 raft_consensus.cc:348] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.669433 8675 raft_consensus.cc:374] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.669544 8675 raft_consensus.cc:729] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.669683 8675 consensus_queue.cc:260] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.669975 8675 ts_tablet_manager.cc:1398] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.746207 8483 raft_consensus.cc:482] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.746338 8483 raft_consensus.cc:504] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.746759 8483 leader_election.cc:288] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.746977 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14165b2b100b4f818ccececc54bd8bad" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:34.747092 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14165b2b100b4f818ccececc54bd8bad" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:34.747175 7139 raft_consensus.cc:2454] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.747238 6878 raft_consensus.cc:2454] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.747488 6945 leader_election.cc:302] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:34.747658 8483 raft_consensus.cc:2781] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.747725 8483 raft_consensus.cc:482] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.747753 8483 raft_consensus.cc:3037] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.748567 8483 raft_consensus.cc:504] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.748867 8483 leader_election.cc:288] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.749157 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14165b2b100b4f818ccececc54bd8bad" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:34.749166 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14165b2b100b4f818ccececc54bd8bad" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:34.749271 6878 raft_consensus.cc:3037] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.749284 7139 raft_consensus.cc:3037] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.750164 6878 raft_consensus.cc:2454] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.750370 7139 raft_consensus.cc:2454] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.750478 6945 leader_election.cc:302] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:34.750830 8483 raft_consensus.cc:2781] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.750909 8483 raft_consensus.cc:686] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:34.751071 8483 consensus_queue.cc:237] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.752053 8483 raft_consensus.cc:482] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.752167 8483 raft_consensus.cc:504] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.752463 8483 leader_election.cc:288] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.752812 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:34.752947 7270 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.752941 6729 catalog_manager.cc:5484] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:34.753309 6944 leader_election.cc:302] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:34.753468 8483 raft_consensus.cc:2781] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.753523 8483 raft_consensus.cc:482] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.753574 8483 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.753840 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:34.753950 6878 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.754419 8483 raft_consensus.cc:504] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.754772 8483 leader_election.cc:288] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:34.755095 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:34.755220 7270 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.755281 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:34.755373 6878 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.756006 6878 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.756026 7270 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.756281 6945 leader_election.cc:302] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:34.756448 8483 raft_consensus.cc:2781] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.756611 8483 raft_consensus.cc:686] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:34.756721 8483 consensus_queue.cc:237] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:34.758354 6729 catalog_manager.cc:5484] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:34.766608 8483 raft_consensus.cc:482] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.766707 8483 raft_consensus.cc:504] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.767095 8483 leader_election.cc:288] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:34.767457 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00569cfef8704ea58ddfeb7bc7c0e20d" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:34.767475 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00569cfef8704ea58ddfeb7bc7c0e20d" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:34.767594 7139 raft_consensus.cc:2454] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.767606 7270 raft_consensus.cc:2454] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:34.767949 6944 leader_election.cc:302] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.768225 8483 raft_consensus.cc:2781] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.768275 8483 raft_consensus.cc:482] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.768340 8483 raft_consensus.cc:3037] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.769186 8483 raft_consensus.cc:504] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.769500 8483 leader_election.cc:288] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:34.769780 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00569cfef8704ea58ddfeb7bc7c0e20d" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:34.769850 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00569cfef8704ea58ddfeb7bc7c0e20d" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:34.769896 7139 raft_consensus.cc:3037] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.769959 7270 raft_consensus.cc:3037] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.770607 7139 raft_consensus.cc:2454] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.770712 7270 raft_consensus.cc:2454] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:34.770917 6944 leader_election.cc:302] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:34.771103 8483 raft_consensus.cc:2781] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.771191 8483 raft_consensus.cc:686] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:34.771502 8483 consensus_queue.cc:237] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:34.773103 6729 catalog_manager.cc:5484] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
W20240501 07:58:34.912984 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.913249 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.916491 8688 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 07:58:35.003424 7270 raft_consensus.cc:1261] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.003902 8483 consensus_queue.cc:1035] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.004205 7139 raft_consensus.cc:1261] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.004701 8483 consensus_queue.cc:1035] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.005373 7271 raft_consensus.cc:1261] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.005365 6878 raft_consensus.cc:1261] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.006304 8597 consensus_queue.cc:1035] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.008694 8597 consensus_queue.cc:1035] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.014322 6877 raft_consensus.cc:1261] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.014838 8597 consensus_queue.cc:1035] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.024132 7139 raft_consensus.cc:1261] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.032331 8724 consensus_queue.cc:1035] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.139 ms
time per row: 0.0625695 ms
I20240501 07:58:35.085529 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:35.090965 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0f4ea5ccba462d9468e3f445569eaa (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.091184 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0f4ea5ccba462d9468e3f445569eaa (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.091357 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0f4ea5ccba462d9468e3f445569eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.091521 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0f4ea5ccba462d9468e3f445569eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.092015 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7eaacb7694552a8fa2f98a5a0c868 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.092260 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7eaacb7694552a8fa2f98a5a0c868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.093283 8674 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:35.091184 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 1671c966902c4eb98b238c4419c525fe (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.093498 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1671c966902c4eb98b238c4419c525fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.093801 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7eaacb7694552a8fa2f98a5a0c868 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.094063 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7eaacb7694552a8fa2f98a5a0c868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.094190 8674 tablet_bootstrap.cc:654] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.094967 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 1671c966902c4eb98b238c4419c525fe (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.095284 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1671c966902c4eb98b238c4419c525fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.095983 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 1671c966902c4eb98b238c4419c525fe (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.096390 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1671c966902c4eb98b238c4419c525fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.096463 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7eaacb7694552a8fa2f98a5a0c868 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.096725 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7eaacb7694552a8fa2f98a5a0c868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.097374 8675 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:35.097486 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0f4ea5ccba462d9468e3f445569eaa (DEFAULT_TABLE table=rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.097698 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0f4ea5ccba462d9468e3f445569eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.098341 8675 tablet_bootstrap.cc:654] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.098647 8672 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:35.099468 8673 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:35.099584 8672 tablet_bootstrap.cc:654] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.099642 8675 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:35.099710 8675 ts_tablet_manager.cc:1367] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:35.100142 8675 raft_consensus.cc:348] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.100255 8675 raft_consensus.cc:374] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.100304 8675 raft_consensus.cc:729] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.100337 8673 tablet_bootstrap.cc:654] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.100435 8675 consensus_queue.cc:260] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.100592 8672 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:35.100668 8672 ts_tablet_manager.cc:1367] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:35.100744 8675 ts_tablet_manager.cc:1398] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.101100 8672 raft_consensus.cc:348] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.101209 8672 raft_consensus.cc:374] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.101240 8672 raft_consensus.cc:729] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.101370 8672 consensus_queue.cc:260] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.101662 8672 ts_tablet_manager.cc:1398] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:35.101763 8674 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:35.101835 8674 ts_tablet_manager.cc:1367] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:58:35.102203 8674 raft_consensus.cc:348] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.102294 8674 raft_consensus.cc:374] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.102329 8674 raft_consensus.cc:729] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.102447 8674 consensus_queue.cc:260] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.102679 8674 ts_tablet_manager.cc:1398] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.102820 8674 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:35.102869 8673 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:35.102941 8673 ts_tablet_manager.cc:1367] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:35.103457 8673 raft_consensus.cc:348] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.103554 8673 raft_consensus.cc:374] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.103583 8673 raft_consensus.cc:729] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.103613 8674 tablet_bootstrap.cc:654] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.103700 8673 consensus_queue.cc:260] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.104012 8673 ts_tablet_manager.cc:1398] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.104876 8675 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:35.105458 8674 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:35.105530 8674 ts_tablet_manager.cc:1367] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:35.105780 8675 tablet_bootstrap.cc:654] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.105929 8674 raft_consensus.cc:348] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.106016 8674 raft_consensus.cc:374] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.106045 8674 raft_consensus.cc:729] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.106159 8674 consensus_queue.cc:260] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.106415 8673 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:35.106436 8674 ts_tablet_manager.cc:1398] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.106573 8674 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:35.106894 8675 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:35.106964 8675 ts_tablet_manager.cc:1367] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.107403 8673 tablet_bootstrap.cc:654] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.107573 8675 raft_consensus.cc:348] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.107671 8675 raft_consensus.cc:374] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.107700 8675 raft_consensus.cc:729] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.107808 8674 tablet_bootstrap.cc:654] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.107837 8675 consensus_queue.cc:260] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.108134 8675 ts_tablet_manager.cc:1398] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.108428 8673 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:35.108512 8673 ts_tablet_manager.cc:1367] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:35.108876 8674 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:35.108956 8674 ts_tablet_manager.cc:1367] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:35.108929 8673 raft_consensus.cc:348] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.109017 8673 raft_consensus.cc:374] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.109061 8673 raft_consensus.cc:729] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.109184 8673 consensus_queue.cc:260] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.109365 8674 raft_consensus.cc:348] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.109455 8674 raft_consensus.cc:374] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.109462 8673 ts_tablet_manager.cc:1398] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.109495 8674 raft_consensus.cc:729] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.109604 8673 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:35.109607 8674 consensus_queue.cc:260] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.109871 8674 ts_tablet_manager.cc:1398] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.110507 8673 tablet_bootstrap.cc:654] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.111482 8673 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:35.111560 8673 ts_tablet_manager.cc:1367] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.112258 8673 raft_consensus.cc:348] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.112357 8673 raft_consensus.cc:374] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.112399 8673 raft_consensus.cc:729] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.112521 8673 consensus_queue.cc:260] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.113461 8673 ts_tablet_manager.cc:1398] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.234549 8662 raft_consensus.cc:482] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.234670 8662 raft_consensus.cc:504] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.235147 8662 leader_election.cc:288] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:35.235525 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1671c966902c4eb98b238c4419c525fe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:35.235522 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1671c966902c4eb98b238c4419c525fe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:35.235669 7141 raft_consensus.cc:2454] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:35.235669 6877 raft_consensus.cc:2454] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:35.235970 7207 leader_election.cc:302] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.236150 8662 raft_consensus.cc:2781] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.236212 8662 raft_consensus.cc:482] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.236240 8662 raft_consensus.cc:3037] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.237052 8662 raft_consensus.cc:504] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.237370 8662 leader_election.cc:288] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:35.237676 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1671c966902c4eb98b238c4419c525fe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:35.237676 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1671c966902c4eb98b238c4419c525fe" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:35.237777 6877 raft_consensus.cc:3037] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.237785 7141 raft_consensus.cc:3037] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.238512 7141 raft_consensus.cc:2454] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:35.238550 6877 raft_consensus.cc:2454] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:35.238775 7206 leader_election.cc:302] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.238947 8662 raft_consensus.cc:2781] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.239069 8662 raft_consensus.cc:686] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:35.239209 8662 consensus_queue.cc:237] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.240878 6732 catalog_manager.cc:5484] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:35.285792 8579 raft_consensus.cc:482] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.285912 8579 raft_consensus.cc:504] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.286340 8579 leader_election.cc:288] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.286633 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:35.286664 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:35.286773 7141 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:35.286793 7272 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:35.287073 6813 leader_election.cc:302] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.287230 8579 raft_consensus.cc:2781] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.287290 8579 raft_consensus.cc:482] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.287341 8579 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.288187 8579 raft_consensus.cc:504] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.288517 8579 leader_election.cc:288] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.288831 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:35.288878 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:35.288945 7272 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.288990 7141 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.289856 7272 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:35.290014 7141 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:35.290271 6813 leader_election.cc:302] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.290417 8579 raft_consensus.cc:2781] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.290496 8579 raft_consensus.cc:686] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:35.290611 8579 consensus_queue.cc:237] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.292366 6732 catalog_manager.cc:5484] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:35.325834 8717 raft_consensus.cc:482] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.325973 8717 raft_consensus.cc:504] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.326364 8717 leader_election.cc:288] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.326694 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7eaacb7694552a8fa2f98a5a0c868" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:35.326699 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7eaacb7694552a8fa2f98a5a0c868" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:35.326817 7141 raft_consensus.cc:2454] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:35.326831 7010 raft_consensus.cc:2454] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:35.327096 6813 leader_election.cc:302] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.327252 8717 raft_consensus.cc:2781] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.327309 8717 raft_consensus.cc:482] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.327345 8717 raft_consensus.cc:3037] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.328349 8717 raft_consensus.cc:504] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.328656 8717 leader_election.cc:288] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.328963 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7eaacb7694552a8fa2f98a5a0c868" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:35.328967 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7eaacb7694552a8fa2f98a5a0c868" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:35.329073 7141 raft_consensus.cc:3037] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.329087 7010 raft_consensus.cc:3037] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.329802 7010 raft_consensus.cc:2454] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:35.329800 7141 raft_consensus.cc:2454] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:35.330056 6813 leader_election.cc:302] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.330209 8717 raft_consensus.cc:2781] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.330293 8717 raft_consensus.cc:686] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:35.330430 8717 consensus_queue.cc:237] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.332321 6734 catalog_manager.cc:5484] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
W20240501 07:58:35.500448 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:35.500713 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:35.503854 8738 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 07:58:35.593640 7141 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.594269 7272 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.594506 7010 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.594997 8579 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.595088 8717 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.595371 8717 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.595935 7141 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.596972 7141 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.600517 8768 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.600988 6877 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.601454 8662 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.602245 8662 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.977 ms
time per row: 0.0609886 ms
I20240501 07:58:35.668056 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:35.673449 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.673779 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280d0dfa82ed4c91831a35e3e10f66d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.673727 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.674001 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3a2b1e39454ba1a538e24e09d543ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.674099 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.674082 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.674306 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48964a6c57584c3eacc6dffac29d77a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.674374 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280d0dfa82ed4c91831a35e3e10f66d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.674813 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.675076 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3a2b1e39454ba1a538e24e09d543ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.675876 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.676127 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48964a6c57584c3eacc6dffac29d77a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.676599 8795 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:35.676752 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.676995 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3a2b1e39454ba1a538e24e09d543ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.677515 8795 tablet_bootstrap.cc:654] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.679438 8796 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:35.679531 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.679682 8794 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:35.679800 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280d0dfa82ed4c91831a35e3e10f66d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.680454 8794 tablet_bootstrap.cc:654] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.680652 8796 tablet_bootstrap.cc:654] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.681355 8794 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:35.681452 8794 ts_tablet_manager.cc:1367] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.681850 8796 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:35.681835 8794 raft_consensus.cc:348] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.681948 8794 raft_consensus.cc:374] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.681996 8800 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:35.681986 8794 raft_consensus.cc:729] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.679536 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:35.682157 8796 ts_tablet_manager.cc:1367] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:35.682191 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48964a6c57584c3eacc6dffac29d77a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:35.682154 8794 consensus_queue.cc:260] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.682515 8794 ts_tablet_manager.cc:1398] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.682662 8794 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:35.682868 8800 tablet_bootstrap.cc:654] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.682894 8796 raft_consensus.cc:348] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.683154 8796 raft_consensus.cc:374] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.683243 8796 raft_consensus.cc:729] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.683439 8796 consensus_queue.cc:260] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.683499 8794 tablet_bootstrap.cc:654] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.683799 8796 ts_tablet_manager.cc:1398] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.684336 8794 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:35.684399 8794 ts_tablet_manager.cc:1367] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.684676 8794 raft_consensus.cc:348] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.684751 8794 raft_consensus.cc:374] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.684772 8794 raft_consensus.cc:729] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.684885 8794 consensus_queue.cc:260] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.685164 8794 ts_tablet_manager.cc:1398] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.685309 8794 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:35.686039 8794 tablet_bootstrap.cc:654] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.686137 8795 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:35.686228 8795 ts_tablet_manager.cc:1367] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:35.686694 8795 raft_consensus.cc:348] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.686794 8795 raft_consensus.cc:374] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.686826 8795 raft_consensus.cc:729] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.686918 8795 consensus_queue.cc:260] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.687168 8795 ts_tablet_manager.cc:1398] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.687297 8795 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:35.687433 8800 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:35.687508 8800 ts_tablet_manager.cc:1367] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:58:35.687970 8795 tablet_bootstrap.cc:654] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.689050 8795 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:35.689126 8795 ts_tablet_manager.cc:1367] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:35.689483 8795 raft_consensus.cc:348] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.689571 8795 raft_consensus.cc:374] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.689600 8795 raft_consensus.cc:729] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.689705 8795 consensus_queue.cc:260] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.689956 8795 ts_tablet_manager.cc:1398] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:35.690131 8795 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:35.690379 8800 raft_consensus.cc:348] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.690500 8800 raft_consensus.cc:374] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.690531 8800 raft_consensus.cc:729] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.690688 8800 consensus_queue.cc:260] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.691128 8800 ts_tablet_manager.cc:1398] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:35.691431 8800 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:35.691640 8795 tablet_bootstrap.cc:654] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.692749 8794 tablet_bootstrap.cc:492] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:35.692831 8794 ts_tablet_manager.cc:1367] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:35.692881 8795 tablet_bootstrap.cc:492] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:35.692946 8795 ts_tablet_manager.cc:1367] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:35.693255 8794 raft_consensus.cc:348] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.693367 8794 raft_consensus.cc:374] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.693339 8795 raft_consensus.cc:348] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.693414 8794 raft_consensus.cc:729] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.693445 8795 raft_consensus.cc:374] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.693468 8795 raft_consensus.cc:729] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.693527 8794 consensus_queue.cc:260] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.693567 8795 consensus_queue.cc:260] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.693817 8795 ts_tablet_manager.cc:1398] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:35.693825 8794 ts_tablet_manager.cc:1398] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.694149 8800 tablet_bootstrap.cc:654] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:35.696370 8800 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:35.696460 8800 ts_tablet_manager.cc:1367] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:35.696889 8800 raft_consensus.cc:348] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.697031 8800 raft_consensus.cc:374] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:35.697073 8800 raft_consensus.cc:729] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:35.697196 8800 consensus_queue.cc:260] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.697444 8800 ts_tablet_manager.cc:1398] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:35.734107 8662 raft_consensus.cc:482] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.734241 8662 raft_consensus.cc:504] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.734644 8662 leader_election.cc:288] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.734946 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3a2b1e39454ba1a538e24e09d543ca" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:35.734959 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3a2b1e39454ba1a538e24e09d543ca" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:35.735148 7139 raft_consensus.cc:2454] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:35.735153 7010 raft_consensus.cc:2454] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:35.735545 7208 leader_election.cc:302] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:35.735724 8662 raft_consensus.cc:2781] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.735796 8662 raft_consensus.cc:482] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.735842 8662 raft_consensus.cc:3037] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.737015 8662 raft_consensus.cc:504] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.737320 8662 leader_election.cc:288] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:35.737648 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3a2b1e39454ba1a538e24e09d543ca" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:35.737746 7139 raft_consensus.cc:3037] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.738466 7139 raft_consensus.cc:2454] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:35.738692 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3a2b1e39454ba1a538e24e09d543ca" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:35.738727 7206 leader_election.cc:302] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:35.738817 7010 raft_consensus.cc:3037] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.738902 8662 raft_consensus.cc:2781] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.739092 8662 raft_consensus.cc:686] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:35.739224 8662 consensus_queue.cc:237] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:35.739825 7010 raft_consensus.cc:2454] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:35.741173 6734 catalog_manager.cc:5484] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:35.826817 8721 raft_consensus.cc:482] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.826929 8721 raft_consensus.cc:504] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.827332 8721 leader_election.cc:288] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:35.827663 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280d0dfa82ed4c91831a35e3e10f66d8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:35.827667 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280d0dfa82ed4c91831a35e3e10f66d8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:35.827790 7272 raft_consensus.cc:2454] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:35.827801 6877 raft_consensus.cc:2454] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:35.828096 6944 leader_election.cc:302] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:35.828261 8721 raft_consensus.cc:2781] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.828323 8721 raft_consensus.cc:482] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.828352 8721 raft_consensus.cc:3037] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.829111 8721 raft_consensus.cc:504] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.829411 8721 leader_election.cc:288] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:35.829676 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280d0dfa82ed4c91831a35e3e10f66d8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:35.829704 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280d0dfa82ed4c91831a35e3e10f66d8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:35.829792 6877 raft_consensus.cc:3037] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.829803 7272 raft_consensus.cc:3037] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.830505 6877 raft_consensus.cc:2454] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:35.830613 7272 raft_consensus.cc:2454] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:35.830801 6945 leader_election.cc:302] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:35.830965 8721 raft_consensus.cc:2781] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.831090 8721 raft_consensus.cc:686] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:35.831405 8721 consensus_queue.cc:237] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:35.833458 6734 catalog_manager.cc:5484] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:35.871826 8721 raft_consensus.cc:482] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.871955 8721 raft_consensus.cc:504] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.872325 8721 leader_election.cc:288] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:35.872706 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48964a6c57584c3eacc6dffac29d77a8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:35.872722 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48964a6c57584c3eacc6dffac29d77a8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:35.872853 6877 raft_consensus.cc:2454] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:35.872864 7272 raft_consensus.cc:2454] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:35.873140 6944 leader_election.cc:302] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:35.873306 8721 raft_consensus.cc:2781] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.873356 8721 raft_consensus.cc:482] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.873383 8721 raft_consensus.cc:3037] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.874172 8721 raft_consensus.cc:504] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.874454 8721 leader_election.cc:288] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:35.874768 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48964a6c57584c3eacc6dffac29d77a8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:35.874814 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48964a6c57584c3eacc6dffac29d77a8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:35.874889 6877 raft_consensus.cc:3037] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.874928 7272 raft_consensus.cc:3037] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.875836 7272 raft_consensus.cc:2454] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:35.875874 6877 raft_consensus.cc:2454] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:35.876092 6944 leader_election.cc:302] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:35.876274 8721 raft_consensus.cc:2781] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.876340 8721 raft_consensus.cc:686] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:35.876479 8721 consensus_queue.cc:237] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:35.878654 6734 catalog_manager.cc:5484] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:36.030752 8810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:36.031075 8810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:36.034106 8810 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 07:58:36.118139 7139 raft_consensus.cc:1261] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.118708 7010 raft_consensus.cc:1261] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.119519 7272 raft_consensus.cc:1261] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.119725 8662 consensus_queue.cc:1035] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.120385 8712 consensus_queue.cc:1035] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.121579 6878 raft_consensus.cc:1261] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.122303 8721 consensus_queue.cc:1035] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.122639 7272 raft_consensus.cc:1261] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.123148 8721 consensus_queue.cc:1035] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.123826 8670 consensus_queue.cc:1035] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.130964 6878 raft_consensus.cc:1261] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.135774 8721 consensus_queue.cc:1035] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.709 ms
time per row: 0.0573544 ms
I20240501 07:58:36.190048 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:36.195870 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 58aab13671d647e485f9be94283b5239 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.196027 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 58aab13671d647e485f9be94283b5239 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.196228 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58aab13671d647e485f9be94283b5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.196336 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58aab13671d647e485f9be94283b5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.198391 8851 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:36.199321 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 58aab13671d647e485f9be94283b5239 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.199321 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 00cac44dfd0f4d97a3056b590748ba58 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.199602 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58aab13671d647e485f9be94283b5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.199630 7119 tablet_service.cc:1467] Processing CreateTablet for tablet 3839146ca1f0452499eb6b5083097d00 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.199726 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cac44dfd0f4d97a3056b590748ba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.199831 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3839146ca1f0452499eb6b5083097d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.200335 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 3839146ca1f0452499eb6b5083097d00 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.200590 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3839146ca1f0452499eb6b5083097d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.201838 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 00cac44dfd0f4d97a3056b590748ba58 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.201885 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 3839146ca1f0452499eb6b5083097d00 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.202109 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cac44dfd0f4d97a3056b590748ba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.202237 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3839146ca1f0452499eb6b5083097d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.202302 8851 tablet_bootstrap.cc:654] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.202718 8852 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:36.203650 8852 tablet_bootstrap.cc:654] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.203891 8855 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:36.205046 8855 tablet_bootstrap.cc:654] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.205142 8852 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:36.205228 8852 ts_tablet_manager.cc:1367] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:36.205237 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 00cac44dfd0f4d97a3056b590748ba58 (DEFAULT_TABLE table=rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.205627 8852 raft_consensus.cc:348] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.205729 8852 raft_consensus.cc:374] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.205760 8852 raft_consensus.cc:729] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.205858 8852 consensus_queue.cc:260] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.206004 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cac44dfd0f4d97a3056b590748ba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.206094 8852 ts_tablet_manager.cc:1398] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.206202 8855 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:36.206254 8852 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:36.206295 8855 ts_tablet_manager.cc:1367] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:36.206707 8855 raft_consensus.cc:348] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.206800 8855 raft_consensus.cc:374] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.206827 8855 raft_consensus.cc:729] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.207135 8852 tablet_bootstrap.cc:654] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.206949 8855 consensus_queue.cc:260] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.207507 8855 ts_tablet_manager.cc:1398] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.207655 8855 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:36.208518 8855 tablet_bootstrap.cc:654] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.209993 8855 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:36.210115 8855 ts_tablet_manager.cc:1367] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:36.210179 8852 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:36.210196 8851 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:36.210266 8852 ts_tablet_manager.cc:1367] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:36.210275 8851 ts_tablet_manager.cc:1367] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:36.210669 8852 raft_consensus.cc:348] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.210671 8855 raft_consensus.cc:348] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.210687 8851 raft_consensus.cc:348] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.210772 8852 raft_consensus.cc:374] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.210772 8851 raft_consensus.cc:374] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.210808 8851 raft_consensus.cc:729] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.210806 8852 raft_consensus.cc:729] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.210896 8855 raft_consensus.cc:374] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.210906 8852 consensus_queue.cc:260] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.210922 8851 consensus_queue.cc:260] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.211037 8855 raft_consensus.cc:729] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.211165 8855 consensus_queue.cc:260] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.211261 8851 ts_tablet_manager.cc:1398] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.211304 8853 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:36.211426 8851 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:36.211503 8855 ts_tablet_manager.cc:1398] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.212239 8851 tablet_bootstrap.cc:654] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.212249 8853 tablet_bootstrap.cc:654] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.214465 8852 ts_tablet_manager.cc:1398] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:36.215266 8851 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:36.215342 8851 ts_tablet_manager.cc:1367] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:36.215693 8851 raft_consensus.cc:348] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.215782 8851 raft_consensus.cc:374] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.215818 8851 raft_consensus.cc:729] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.215935 8851 consensus_queue.cc:260] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.216188 8851 ts_tablet_manager.cc:1398] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.216692 8851 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:36.218577 8851 tablet_bootstrap.cc:654] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.221132 8851 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:36.221220 8851 ts_tablet_manager.cc:1367] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:36.221608 8851 raft_consensus.cc:348] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.221724 8851 raft_consensus.cc:374] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.221760 8851 raft_consensus.cc:729] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.221895 8851 consensus_queue.cc:260] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.222137 8851 ts_tablet_manager.cc:1398] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.223671 8853 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:36.223762 8853 ts_tablet_manager.cc:1367] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 07:58:36.224205 8853 raft_consensus.cc:348] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.224375 8853 raft_consensus.cc:374] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.224423 8853 raft_consensus.cc:729] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.224581 8853 consensus_queue.cc:260] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.224870 8853 ts_tablet_manager.cc:1398] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:36.225093 8853 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:36.225883 8853 tablet_bootstrap.cc:654] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.226871 8853 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:36.226974 8853 ts_tablet_manager.cc:1367] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:36.228160 8853 raft_consensus.cc:348] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.228261 8853 raft_consensus.cc:374] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.228299 8853 raft_consensus.cc:729] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.228410 8853 consensus_queue.cc:260] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.228758 8853 ts_tablet_manager.cc:1398] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:36.235531 8579 raft_consensus.cc:482] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:36.235630 8579 raft_consensus.cc:504] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.236004 8579 leader_election.cc:288] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.236411 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3839146ca1f0452499eb6b5083097d00" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:36.236433 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3839146ca1f0452499eb6b5083097d00" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:36.236549 7271 raft_consensus.cc:2454] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:36.236549 7139 raft_consensus.cc:2454] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:36.236804 6813 leader_election.cc:302] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:36.236950 8579 raft_consensus.cc:2781] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:36.237030 8579 raft_consensus.cc:482] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:36.237095 8579 raft_consensus.cc:3037] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.238013 8579 raft_consensus.cc:504] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.238301 8579 leader_election.cc:288] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.238576 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3839146ca1f0452499eb6b5083097d00" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:36.238615 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3839146ca1f0452499eb6b5083097d00" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:36.238695 7139 raft_consensus.cc:3037] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.238715 7271 raft_consensus.cc:3037] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.239694 7271 raft_consensus.cc:2454] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:36.239694 7139 raft_consensus.cc:2454] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:36.240003 6813 leader_election.cc:302] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:36.240159 8579 raft_consensus.cc:2781] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.240245 8579 raft_consensus.cc:686] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:36.240391 8579 consensus_queue.cc:237] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.242043 6734 catalog_manager.cc:5484] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:36.293689 8721 raft_consensus.cc:482] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:36.293819 8721 raft_consensus.cc:504] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.294245 8721 leader_election.cc:288] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:36.294591 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58aab13671d647e485f9be94283b5239" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:36.294603 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58aab13671d647e485f9be94283b5239" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:36.294728 7139 raft_consensus.cc:2454] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:36.294728 6878 raft_consensus.cc:2454] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:36.294999 6944 leader_election.cc:302] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.295184 8721 raft_consensus.cc:2781] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:36.295253 8721 raft_consensus.cc:482] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:36.295297 8721 raft_consensus.cc:3037] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.296238 8721 raft_consensus.cc:504] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.296582 8721 leader_election.cc:288] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:36.296888 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58aab13671d647e485f9be94283b5239" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:36.296993 6878 raft_consensus.cc:3037] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.296984 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58aab13671d647e485f9be94283b5239" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:36.297111 7139 raft_consensus.cc:3037] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.297899 7139 raft_consensus.cc:2454] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:36.297978 6878 raft_consensus.cc:2454] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:36.298254 6944 leader_election.cc:302] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.298437 8712 raft_consensus.cc:2781] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.298547 8712 raft_consensus.cc:686] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:36.298686 8712 consensus_queue.cc:237] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.300827 6734 catalog_manager.cc:5484] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:36.476329 8657 raft_consensus.cc:482] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:36.476456 8657 raft_consensus.cc:504] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.476864 8657 leader_election.cc:288] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.477285 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cac44dfd0f4d97a3056b590748ba58" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:36.477285 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cac44dfd0f4d97a3056b590748ba58" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:36.477429 7010 raft_consensus.cc:2454] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.477429 7271 raft_consensus.cc:2454] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.477742 7077 leader_election.cc:302] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.477900 8657 raft_consensus.cc:2781] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:36.477972 8657 raft_consensus.cc:482] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:36.478005 8657 raft_consensus.cc:3037] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.478940 8657 raft_consensus.cc:504] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.479303 8657 leader_election.cc:288] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.479581 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cac44dfd0f4d97a3056b590748ba58" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:36.479583 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cac44dfd0f4d97a3056b590748ba58" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:36.479692 7010 raft_consensus.cc:3037] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.479692 7271 raft_consensus.cc:3037] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.481000 7271 raft_consensus.cc:2454] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.481000 7010 raft_consensus.cc:2454] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.481582 7075 leader_election.cc:302] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.481864 8657 raft_consensus.cc:2781] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.482237 8657 raft_consensus.cc:686] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:36.482528 8657 consensus_queue.cc:237] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.484758 6734 catalog_manager.cc:5484] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:36.614421 8868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:36.614665 8868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:36.617712 8868 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:36.637287 8579 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240501 07:58:36.640959 8579 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.699020 6878 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.699128 7010 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.699419 8721 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.699493 8657 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.700284 7139 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.700673 7272 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:36.703491 8712 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:36.704084 8657 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.849 ms
time per row: 0.0519245 ms
I20240501 07:58:36.765215 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:36.770480 7120 tablet_service.cc:1467] Processing CreateTablet for tablet e05498632f0945c29f723a4613a26a04 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.770789 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05498632f0945c29f723a4613a26a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.771324 7119 tablet_service.cc:1467] Processing CreateTablet for tablet daa83d6382e340d28e60df08fd470d52 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.771350 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3d410c640243848a478aa2605c1bf0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.771569 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa83d6382e340d28e60df08fd470d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.771569 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3d410c640243848a478aa2605c1bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.772776 8931 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:36.772944 6857 tablet_service.cc:1467] Processing CreateTablet for tablet e05498632f0945c29f723a4613a26a04 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.773193 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05498632f0945c29f723a4613a26a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.773833 8931 tablet_bootstrap.cc:654] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.773969 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3d410c640243848a478aa2605c1bf0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.774245 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3d410c640243848a478aa2605c1bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.775465 8931 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:36.775521 8931 ts_tablet_manager.cc:1367] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:36.775789 8931 raft_consensus.cc:348] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.775862 8931 raft_consensus.cc:374] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.775895 8931 raft_consensus.cc:729] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.776014 8931 consensus_queue.cc:260] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.776080 8934 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:36.776185 7250 tablet_service.cc:1467] Processing CreateTablet for tablet daa83d6382e340d28e60df08fd470d52 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.776299 8931 ts_tablet_manager.cc:1398] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.776418 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa83d6382e340d28e60df08fd470d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.776434 8931 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:36.776880 8934 tablet_bootstrap.cc:654] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.777279 8931 tablet_bootstrap.cc:654] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.778002 8934 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:36.778059 8934 ts_tablet_manager.cc:1367] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:36.778441 8934 raft_consensus.cc:348] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.778546 8934 raft_consensus.cc:374] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.778584 8934 raft_consensus.cc:729] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.778736 8934 consensus_queue.cc:260] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.779264 8934 ts_tablet_manager.cc:1398] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.779415 8934 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:36.779372 6989 tablet_service.cc:1467] Processing CreateTablet for tablet e05498632f0945c29f723a4613a26a04 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.779503 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3d410c640243848a478aa2605c1bf0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.779644 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05498632f0945c29f723a4613a26a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.780180 8934 tablet_bootstrap.cc:654] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.780519 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3d410c640243848a478aa2605c1bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.781790 8938 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:36.781940 6988 tablet_service.cc:1467] Processing CreateTablet for tablet daa83d6382e340d28e60df08fd470d52 (DEFAULT_TABLE table=rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:36.782186 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa83d6382e340d28e60df08fd470d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:36.782598 8938 tablet_bootstrap.cc:654] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.784003 8938 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:36.784072 8938 ts_tablet_manager.cc:1367] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:36.784461 8932 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:36.784497 8938 raft_consensus.cc:348] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.784637 8938 raft_consensus.cc:374] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.784682 8938 raft_consensus.cc:729] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.784785 8938 consensus_queue.cc:260] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.785020 8938 ts_tablet_manager.cc:1398] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:36.785162 8938 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:36.785858 8932 tablet_bootstrap.cc:654] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.785921 8938 tablet_bootstrap.cc:654] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.786870 8932 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:36.786953 8932 ts_tablet_manager.cc:1367] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:36.787389 8932 raft_consensus.cc:348] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.787501 8932 raft_consensus.cc:374] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.787540 8932 raft_consensus.cc:729] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.787662 8932 consensus_queue.cc:260] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.787915 8932 ts_tablet_manager.cc:1398] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.788067 8932 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:36.788924 8932 tablet_bootstrap.cc:654] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.789958 8932 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:36.790038 8932 ts_tablet_manager.cc:1367] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:36.790426 8932 raft_consensus.cc:348] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.790499 8932 raft_consensus.cc:374] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.790519 8932 raft_consensus.cc:729] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.790620 8932 consensus_queue.cc:260] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.790882 8932 ts_tablet_manager.cc:1398] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.791419 8931 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:36.791533 8931 ts_tablet_manager.cc:1367] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:36.791947 8931 raft_consensus.cc:348] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.792064 8931 raft_consensus.cc:374] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.792100 8931 raft_consensus.cc:729] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.792241 8931 consensus_queue.cc:260] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.792558 8931 ts_tablet_manager.cc:1398] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:36.793241 8934 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:36.793332 8934 ts_tablet_manager.cc:1367] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:58:36.793743 8934 raft_consensus.cc:348] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.793828 8934 raft_consensus.cc:374] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.793851 8934 raft_consensus.cc:729] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.793946 8934 consensus_queue.cc:260] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.794169 8934 ts_tablet_manager.cc:1398] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.794876 8938 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:36.794953 8938 ts_tablet_manager.cc:1367] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 07:58:36.795377 8938 raft_consensus.cc:348] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.795483 8938 raft_consensus.cc:374] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.795511 8938 raft_consensus.cc:729] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.795603 8938 consensus_queue.cc:260] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:36.795815 8938 ts_tablet_manager.cc:1398] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.795944 8938 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:36.796763 8938 tablet_bootstrap.cc:654] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:36.799301 8938 tablet_bootstrap.cc:492] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:36.799381 8938 ts_tablet_manager.cc:1367] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:36.799798 8938 raft_consensus.cc:348] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.799911 8938 raft_consensus.cc:374] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:36.799953 8938 raft_consensus.cc:729] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:36.800068 8938 consensus_queue.cc:260] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.800318 8938 ts_tablet_manager.cc:1398] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:36.814971 8909 raft_consensus.cc:482] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:36.815093 8909 raft_consensus.cc:504] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.815541 8909 leader_election.cc:288] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.815807 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa83d6382e340d28e60df08fd470d52" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:36.815943 7010 raft_consensus.cc:2454] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.816162 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa83d6382e340d28e60df08fd470d52" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:36.816279 7077 leader_election.cc:302] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.816303 7272 raft_consensus.cc:2454] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.816454 8909 raft_consensus.cc:2781] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:36.816529 8909 raft_consensus.cc:482] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:36.816576 8909 raft_consensus.cc:3037] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.817659 8909 raft_consensus.cc:504] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.818063 8909 leader_election.cc:288] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:36.818295 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa83d6382e340d28e60df08fd470d52" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:36.818398 7010 raft_consensus.cc:3037] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.818441 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa83d6382e340d28e60df08fd470d52" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:36.818534 7272 raft_consensus.cc:3037] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.819541 7272 raft_consensus.cc:2454] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.819557 7010 raft_consensus.cc:2454] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.819826 7075 leader_election.cc:302] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.820068 8909 raft_consensus.cc:2781] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.820183 8909 raft_consensus.cc:686] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:36.820334 8909 consensus_queue.cc:237] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:36.822250 6734 catalog_manager.cc:5484] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:36.891650 8909 raft_consensus.cc:482] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:36.891762 8909 raft_consensus.cc:504] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.892249 8909 leader_election.cc:288] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:36.892539 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05498632f0945c29f723a4613a26a04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:36.892549 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05498632f0945c29f723a4613a26a04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:36.892663 7010 raft_consensus.cc:2454] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.892676 6878 raft_consensus.cc:2454] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:36.892915 7077 leader_election.cc:302] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.893155 8909 raft_consensus.cc:2781] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:36.893203 8909 raft_consensus.cc:482] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:36.893255 8909 raft_consensus.cc:3037] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.894035 8909 raft_consensus.cc:504] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.894341 8909 leader_election.cc:288] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:36.894614 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05498632f0945c29f723a4613a26a04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:36.894629 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05498632f0945c29f723a4613a26a04" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:36.894739 6878 raft_consensus.cc:3037] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.894754 7010 raft_consensus.cc:3037] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:36.895570 6878 raft_consensus.cc:2454] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.895614 7010 raft_consensus.cc:2454] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:36.895831 7076 leader_election.cc:302] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:36.895979 8909 raft_consensus.cc:2781] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:36.896077 8909 raft_consensus.cc:686] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:36.896214 8909 consensus_queue.cc:237] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:36.898295 6734 catalog_manager.cc:5484] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:37.023293 8712 raft_consensus.cc:482] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.023437 8712 raft_consensus.cc:504] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.023823 8712 leader_election.cc:288] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:37.024248 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3d410c640243848a478aa2605c1bf0" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:37.024272 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3d410c640243848a478aa2605c1bf0" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:37.024385 6877 raft_consensus.cc:2454] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:37.024530 7272 raft_consensus.cc:2454] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:37.024690 6945 leader_election.cc:302] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:37.024873 8712 raft_consensus.cc:2781] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.024919 8712 raft_consensus.cc:482] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.024951 8712 raft_consensus.cc:3037] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.026070 8712 raft_consensus.cc:504] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.026372 8712 leader_election.cc:288] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:37.026651 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3d410c640243848a478aa2605c1bf0" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:37.026696 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3d410c640243848a478aa2605c1bf0" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:37.026748 7272 raft_consensus.cc:3037] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.026804 6877 raft_consensus.cc:3037] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.027549 7272 raft_consensus.cc:2454] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:37.027714 6877 raft_consensus.cc:2454] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:37.027792 6944 leader_election.cc:302] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:37.028110 8712 raft_consensus.cc:2781] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.028208 8712 raft_consensus.cc:686] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:37.028331 8712 consensus_queue.cc:237] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.030334 6734 catalog_manager.cc:5484] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:37.181244 8947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:37.181475 8947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:37.184851 8947 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240501 07:58:37.243573 8909 consensus_queue.cc:1035] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.250355 8866 consensus_queue.cc:1035] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.281265 7010 raft_consensus.cc:1261] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.281641 8909 consensus_queue.cc:1035] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.283131 6878 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.283522 7271 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.283773 8712 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.283861 6877 raft_consensus.cc:1261] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.284268 8909 consensus_queue.cc:1035] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.285107 8712 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.293888 8974 mvcc.cc:205] Tried to move back new op lower bound from 7022798099579101184 to 7022798098008190976. Current Snapshot: MvccSnapshot[applied={T|T < 7022798099579101184}]
I20240501 07:58:37.298568 8984 mvcc.cc:205] Tried to move back new op lower bound from 7022798099579101184 to 7022798098008190976. Current Snapshot: MvccSnapshot[applied={T|T < 7022798099579101184}]
I20240501 07:58:37.311853 8972 mvcc.cc:205] Tried to move back new op lower bound from 7022798099585847296 to 7022798098549252096. Current Snapshot: MvccSnapshot[applied={T|T < 7022798099585847296}]
INSERT report
rows total: 2000
time total: 114.306 ms
time per row: 0.057153 ms
I20240501 07:58:37.356854 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:37.362644 6857 tablet_service.cc:1467] Processing CreateTablet for tablet ab7dcdd38eff43fe9fff9081ee644c8d (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.362988 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7dcdd38eff43fe9fff9081ee644c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.362985 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 2eec88312feb44feb70401fe29a3bb83 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.363278 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eec88312feb44feb70401fe29a3bb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.363546 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 2304f5e5d09a47b793cbb4f60b255525 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.363997 7119 tablet_service.cc:1467] Processing CreateTablet for tablet ab7dcdd38eff43fe9fff9081ee644c8d (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.364266 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7dcdd38eff43fe9fff9081ee644c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.364425 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 2eec88312feb44feb70401fe29a3bb83 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.364822 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 2304f5e5d09a47b793cbb4f60b255525 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.364936 7250 tablet_service.cc:1467] Processing CreateTablet for tablet ab7dcdd38eff43fe9fff9081ee644c8d (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.364935 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 2eec88312feb44feb70401fe29a3bb83 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.365278 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7dcdd38eff43fe9fff9081ee644c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.365432 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eec88312feb44feb70401fe29a3bb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.365474 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 2304f5e5d09a47b793cbb4f60b255525 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.366086 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304f5e5d09a47b793cbb4f60b255525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.366556 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eec88312feb44feb70401fe29a3bb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.368602 9003 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:37.368809 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304f5e5d09a47b793cbb4f60b255525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.369750 9003 tablet_bootstrap.cc:654] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.371484 9003 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:37.371568 9003 ts_tablet_manager.cc:1367] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:37.372057 9003 raft_consensus.cc:348] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.372165 9003 raft_consensus.cc:374] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.372205 9003 raft_consensus.cc:729] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.372341 9003 consensus_queue.cc:260] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.372800 9003 ts_tablet_manager.cc:1398] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.373102 9003 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:37.374292 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304f5e5d09a47b793cbb4f60b255525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.374655 9006 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:37.375669 9006 tablet_bootstrap.cc:654] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.376926 9008 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:37.377928 9008 tablet_bootstrap.cc:654] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.379652 9008 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:37.379797 9008 ts_tablet_manager.cc:1367] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:37.380136 9003 tablet_bootstrap.cc:654] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.380477 9008 raft_consensus.cc:348] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.380664 9008 raft_consensus.cc:374] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.380724 9008 raft_consensus.cc:729] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.380873 9008 consensus_queue.cc:260] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.381300 9008 ts_tablet_manager.cc:1398] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:37.381335 9003 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:37.381408 9003 ts_tablet_manager.cc:1367] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:37.381511 9008 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:37.381860 9003 raft_consensus.cc:348] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.381975 9003 raft_consensus.cc:374] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.382011 9003 raft_consensus.cc:729] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.382158 9003 consensus_queue.cc:260] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.382462 9008 tablet_bootstrap.cc:654] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.383339 9003 ts_tablet_manager.cc:1398] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.383538 9003 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:37.384362 9003 tablet_bootstrap.cc:654] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.385640 9003 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:37.385731 9003 ts_tablet_manager.cc:1367] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:37.386181 9003 raft_consensus.cc:348] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.386298 9003 raft_consensus.cc:374] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.386340 9003 raft_consensus.cc:729] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.386466 9003 consensus_queue.cc:260] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.386804 9003 ts_tablet_manager.cc:1398] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.387163 9004 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:37.387576 9008 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:37.387672 9008 ts_tablet_manager.cc:1367] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:58:37.388034 9004 tablet_bootstrap.cc:654] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.388186 9008 raft_consensus.cc:348] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.388296 9008 raft_consensus.cc:374] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.388347 9008 raft_consensus.cc:729] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.388486 9008 consensus_queue.cc:260] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.389091 9004 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:37.389173 9004 ts_tablet_manager.cc:1367] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.389453 9008 ts_tablet_manager.cc:1398] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:37.389642 9008 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:37.389643 9004 raft_consensus.cc:348] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.389744 9004 raft_consensus.cc:374] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.389776 9004 raft_consensus.cc:729] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.389900 9004 consensus_queue.cc:260] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.390515 9004 ts_tablet_manager.cc:1398] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.390565 9008 tablet_bootstrap.cc:654] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.390707 9004 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:37.391644 9004 tablet_bootstrap.cc:654] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.392441 9008 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:37.392532 9008 ts_tablet_manager.cc:1367] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:37.392969 9008 raft_consensus.cc:348] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.393085 9008 raft_consensus.cc:374] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.393121 9008 raft_consensus.cc:729] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.393255 9008 consensus_queue.cc:260] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.393543 9006 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:37.393623 9008 ts_tablet_manager.cc:1398] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:37.393648 9006 ts_tablet_manager.cc:1367] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:58:37.394059 9006 raft_consensus.cc:348] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.394160 9006 raft_consensus.cc:374] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.394194 9006 raft_consensus.cc:729] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.394322 9006 consensus_queue.cc:260] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.394589 9006 ts_tablet_manager.cc:1398] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.396378 9004 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:37.396485 9004 ts_tablet_manager.cc:1367] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:37.396991 9004 raft_consensus.cc:348] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.397080 9004 raft_consensus.cc:374] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.397114 9004 raft_consensus.cc:729] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.397254 9004 consensus_queue.cc:260] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.398353 9004 ts_tablet_manager.cc:1398] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.495352 8717 raft_consensus.cc:482] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.495478 8717 raft_consensus.cc:504] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.495868 8717 leader_election.cc:288] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:37.496174 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304f5e5d09a47b793cbb4f60b255525" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:37.496239 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304f5e5d09a47b793cbb4f60b255525" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:37.496296 7140 raft_consensus.cc:2454] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:37.496377 7009 raft_consensus.cc:2454] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:37.496623 6813 leader_election.cc:302] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:37.496803 8717 raft_consensus.cc:2781] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.496888 8717 raft_consensus.cc:482] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.496932 8717 raft_consensus.cc:3037] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.497875 8717 raft_consensus.cc:504] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.498221 8717 leader_election.cc:288] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:37.498589 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304f5e5d09a47b793cbb4f60b255525" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:37.498714 7009 raft_consensus.cc:3037] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.499645 7009 raft_consensus.cc:2454] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:37.499893 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2304f5e5d09a47b793cbb4f60b255525" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:37.499951 6815 leader_election.cc:302] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:37.500022 7140 raft_consensus.cc:3037] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.500138 8717 raft_consensus.cc:2781] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.500360 8717 raft_consensus.cc:686] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:37.500517 8717 consensus_queue.cc:237] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.501068 7140 raft_consensus.cc:2454] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:37.502583 6729 catalog_manager.cc:5484] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:37.516887 8998 raft_consensus.cc:482] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.516988 8998 raft_consensus.cc:504] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.517400 8998 leader_election.cc:288] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:37.517868 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eec88312feb44feb70401fe29a3bb83" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:37.518015 6877 raft_consensus.cc:2454] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:37.518020 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eec88312feb44feb70401fe29a3bb83" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:37.518151 7271 raft_consensus.cc:2454] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:37.518311 7076 leader_election.cc:302] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:37.518493 8998 raft_consensus.cc:2781] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.518545 8998 raft_consensus.cc:482] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.518594 8998 raft_consensus.cc:3037] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.519505 8998 raft_consensus.cc:504] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.519829 8998 leader_election.cc:288] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:37.520138 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eec88312feb44feb70401fe29a3bb83" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:37.520138 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eec88312feb44feb70401fe29a3bb83" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:37.520255 7271 raft_consensus.cc:3037] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.520255 6877 raft_consensus.cc:3037] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.521253 7271 raft_consensus.cc:2454] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:37.521252 6877 raft_consensus.cc:2454] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:37.521567 7076 leader_election.cc:302] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:37.521728 8998 raft_consensus.cc:2781] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.521821 8998 raft_consensus.cc:686] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:37.521958 8998 consensus_queue.cc:237] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:37.523845 6729 catalog_manager.cc:5484] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:37.524303 8670 raft_consensus.cc:482] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.524389 8670 raft_consensus.cc:504] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.524746 8670 leader_election.cc:288] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:37.525060 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7dcdd38eff43fe9fff9081ee644c8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:37.525179 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7dcdd38eff43fe9fff9081ee644c8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:37.525200 7140 raft_consensus.cc:2454] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:37.525287 6877 raft_consensus.cc:2454] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:37.525449 7206 leader_election.cc:302] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:37.525589 8670 raft_consensus.cc:2781] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.525655 8670 raft_consensus.cc:482] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.525693 8670 raft_consensus.cc:3037] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.526556 8670 raft_consensus.cc:504] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.526839 8670 leader_election.cc:288] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:37.527168 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7dcdd38eff43fe9fff9081ee644c8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:37.527187 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7dcdd38eff43fe9fff9081ee644c8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:37.527268 7140 raft_consensus.cc:3037] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.527271 6877 raft_consensus.cc:3037] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.527942 7140 raft_consensus.cc:2454] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:37.527954 6877 raft_consensus.cc:2454] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:37.528167 7206 leader_election.cc:302] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:37.528324 8670 raft_consensus.cc:2781] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.528414 8670 raft_consensus.cc:686] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:37.528611 8670 consensus_queue.cc:237] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.530508 6729 catalog_manager.cc:5484] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:37.663949 9021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:37.664228 9021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:37.667307 9021 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 07:58:37.760586 6877 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.760987 8866 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.761802 7140 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.762113 7270 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.762122 7009 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.762629 6878 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.763236 8670 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.763666 8717 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.764243 8670 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.765564 8866 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.771736 7139 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:37.772617 8717 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:37.775465 8971 mvcc.cc:205] Tried to move back new op lower bound from 7022798101559083008 to 7022798100570984448. Current Snapshot: MvccSnapshot[applied={T|T < 7022798101542400000}]
INSERT report
rows total: 2000
time total: 125.362 ms
time per row: 0.0626809 ms
I20240501 07:58:37.834870 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:37.840201 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.840516 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eb30b03de54bccb36f5b8093abf4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.840828 6989 tablet_service.cc:1467] Processing CreateTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.840828 6990 tablet_service.cc:1467] Processing CreateTablet for tablet c1889e43466f46018b95673d30fdd118 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.841130 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1889e43466f46018b95673d30fdd118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.841260 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.842299 9006 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:37.843232 9006 tablet_bootstrap.cc:654] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.843370 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.843410 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.843627 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eb30b03de54bccb36f5b8093abf4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.843765 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.843963 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.844182 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eb30b03de54bccb36f5b8093abf4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.844220 9006 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:37.844290 9006 ts_tablet_manager.cc:1367] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.844700 9006 raft_consensus.cc:348] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.844807 9006 raft_consensus.cc:374] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.844838 9006 raft_consensus.cc:729] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.844975 9006 consensus_queue.cc:260] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.845263 9006 ts_tablet_manager.cc:1398] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.845402 9006 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:37.845791 9008 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:37.845863 9004 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:37.845973 6857 tablet_service.cc:1467] Processing CreateTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.846184 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.846648 9008 tablet_bootstrap.cc:654] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.846637 9004 tablet_bootstrap.cc:654] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.848227 7252 tablet_service.cc:1467] Processing CreateTablet for tablet c1889e43466f46018b95673d30fdd118 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.848439 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1889e43466f46018b95673d30fdd118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.849138 9006 tablet_bootstrap.cc:654] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.850080 9004 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:37.850076 9006 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:37.850147 9004 ts_tablet_manager.cc:1367] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:37.850147 9006 ts_tablet_manager.cc:1367] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:37.850453 7121 tablet_service.cc:1467] Processing CreateTablet for tablet c1889e43466f46018b95673d30fdd118 (DEFAULT_TABLE table=rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:37.850503 9006 raft_consensus.cc:348] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.850513 9004 raft_consensus.cc:348] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.850596 9006 raft_consensus.cc:374] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.850597 9004 raft_consensus.cc:374] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.850628 9006 raft_consensus.cc:729] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.850661 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1889e43466f46018b95673d30fdd118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:37.850658 9004 raft_consensus.cc:729] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.850750 9006 consensus_queue.cc:260] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.850757 9004 consensus_queue.cc:260] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.851024 9004 ts_tablet_manager.cc:1398] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.851169 9004 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:37.851985 9004 tablet_bootstrap.cc:654] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.852542 9003 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:37.852736 9006 ts_tablet_manager.cc:1398] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:37.852874 9006 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:37.853343 9003 tablet_bootstrap.cc:654] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.853753 9006 tablet_bootstrap.cc:654] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.854007 9004 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:37.854079 9004 ts_tablet_manager.cc:1367] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:37.854271 9008 tablet_bootstrap.cc:492] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:37.854349 9008 ts_tablet_manager.cc:1367] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:58:37.854460 9004 raft_consensus.cc:348] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.854547 9004 raft_consensus.cc:374] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.854578 9004 raft_consensus.cc:729] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.854691 9004 consensus_queue.cc:260] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.854794 9006 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:37.854877 9006 ts_tablet_manager.cc:1367] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:37.854830 9008 raft_consensus.cc:348] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.854920 9008 raft_consensus.cc:374] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.854940 9004 ts_tablet_manager.cc:1398] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.854951 9008 raft_consensus.cc:729] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.855111 9004 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:37.855114 9008 consensus_queue.cc:260] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.855392 9008 ts_tablet_manager.cc:1398] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.855386 9006 raft_consensus.cc:348] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.855477 9006 raft_consensus.cc:374] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.855510 9006 raft_consensus.cc:729] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.855557 9008 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:37.855615 9006 consensus_queue.cc:260] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.855898 9006 ts_tablet_manager.cc:1398] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.856415 9008 tablet_bootstrap.cc:654] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.856981 9004 tablet_bootstrap.cc:654] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:37.857856 9008 tablet_bootstrap.cc:492] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:37.857934 9008 ts_tablet_manager.cc:1367] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:37.858239 9004 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:37.858364 9004 ts_tablet_manager.cc:1367] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:37.858335 9008 raft_consensus.cc:348] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.858422 9008 raft_consensus.cc:374] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.858477 9008 raft_consensus.cc:729] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.858611 9008 consensus_queue.cc:260] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:37.858937 9008 ts_tablet_manager.cc:1398] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.858960 9003 tablet_bootstrap.cc:492] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:37.859079 9003 ts_tablet_manager.cc:1367] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:37.859109 9004 raft_consensus.cc:348] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.859208 9004 raft_consensus.cc:374] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.859274 9004 raft_consensus.cc:729] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.859391 9004 consensus_queue.cc:260] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.859573 9003 raft_consensus.cc:348] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.859673 9003 raft_consensus.cc:374] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:37.859690 9004 ts_tablet_manager.cc:1398] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:37.859717 9003 raft_consensus.cc:729] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:37.859885 9003 consensus_queue.cc:260] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:37.860232 9003 ts_tablet_manager.cc:1398] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:37.928717 9059 raft_consensus.cc:482] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:37.928874 9059 raft_consensus.cc:504] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.929313 9059 leader_election.cc:288] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:37.929587 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eb30b03de54bccb36f5b8093abf4e3" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:37.929677 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eb30b03de54bccb36f5b8093abf4e3" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:37.929726 7009 raft_consensus.cc:2454] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:37.929838 7270 raft_consensus.cc:2454] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:37.930130 6813 leader_election.cc:302] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:37.930285 9059 raft_consensus.cc:2781] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:37.930344 9059 raft_consensus.cc:482] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:37.930371 9059 raft_consensus.cc:3037] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.931255 9059 raft_consensus.cc:504] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.931551 9059 leader_election.cc:288] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:37.931813 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eb30b03de54bccb36f5b8093abf4e3" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:37.931855 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eb30b03de54bccb36f5b8093abf4e3" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:37.931964 7009 raft_consensus.cc:3037] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.931975 7270 raft_consensus.cc:3037] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:37.932891 7009 raft_consensus.cc:2454] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:37.932909 7270 raft_consensus.cc:2454] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:37.933133 6815 leader_election.cc:302] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:37.933298 9059 raft_consensus.cc:2781] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:37.933382 9059 raft_consensus.cc:686] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:37.933512 9059 consensus_queue.cc:237] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:37.935303 6730 catalog_manager.cc:5484] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:38.001919 8980 raft_consensus.cc:482] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.002044 8980 raft_consensus.cc:504] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.002427 8980 leader_election.cc:288] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:38.002699 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1889e43466f46018b95673d30fdd118" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:38.002735 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1889e43466f46018b95673d30fdd118" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:38.002852 7139 raft_consensus.cc:2454] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:38.002843 7270 raft_consensus.cc:2454] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:38.003091 6944 leader_election.cc:302] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.003233 8980 raft_consensus.cc:2781] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:38.003286 8980 raft_consensus.cc:482] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:38.003333 8980 raft_consensus.cc:3037] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.004185 8980 raft_consensus.cc:504] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.004485 8980 leader_election.cc:288] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:38.004740 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1889e43466f46018b95673d30fdd118" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:38.004776 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1889e43466f46018b95673d30fdd118" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:38.004868 7139 raft_consensus.cc:3037] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.004855 7270 raft_consensus.cc:3037] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.005779 7139 raft_consensus.cc:2454] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:38.005779 7270 raft_consensus.cc:2454] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:38.006042 6944 leader_election.cc:302] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.006183 8980 raft_consensus.cc:2781] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.006274 8980 raft_consensus.cc:686] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:38.006428 8980 consensus_queue.cc:237] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.008186 6730 catalog_manager.cc:5484] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:38.152772 9059 raft_consensus.cc:482] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.152884 9059 raft_consensus.cc:504] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:38.153228 9059 leader_election.cc:288] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:38.153533 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c3cb8aafcd4ab7be9018e6c63f0e50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:38.153544 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c3cb8aafcd4ab7be9018e6c63f0e50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:38.153647 7270 raft_consensus.cc:2454] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:38.153656 7009 raft_consensus.cc:2454] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:38.153911 6815 leader_election.cc:302] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:38.154064 9059 raft_consensus.cc:2781] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:38.154116 9059 raft_consensus.cc:482] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:38.154167 9059 raft_consensus.cc:3037] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.154927 9059 raft_consensus.cc:504] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:38.155244 9059 leader_election.cc:288] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:38.155495 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c3cb8aafcd4ab7be9018e6c63f0e50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:38.155527 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c3cb8aafcd4ab7be9018e6c63f0e50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:38.155603 7009 raft_consensus.cc:3037] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.155620 7270 raft_consensus.cc:3037] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.156474 7009 raft_consensus.cc:2454] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:38.156474 7270 raft_consensus.cc:2454] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:38.156728 6813 leader_election.cc:302] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:38.156931 9059 raft_consensus.cc:2781] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.156996 9059 raft_consensus.cc:686] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:38.157287 9059 consensus_queue.cc:237] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:38.158936 6730 catalog_manager.cc:5484] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:38.321897 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:38.322119 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:38.325409 9075 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 07:58:38.408766 7270 raft_consensus.cc:1261] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.409664 7009 raft_consensus.cc:1261] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.410476 9059 consensus_queue.cc:1035] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.411214 9017 consensus_queue.cc:1035] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.413558 7270 raft_consensus.cc:1261] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.413962 9059 consensus_queue.cc:1035] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.417169 7010 raft_consensus.cc:1261] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.417987 9017 consensus_queue.cc:1035] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.449541 7139 raft_consensus.cc:1261] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.449887 7271 raft_consensus.cc:1261] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.452662 8721 consensus_queue.cc:1035] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.452996 8721 consensus_queue.cc:1035] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.246 ms
time per row: 0.063123 ms
I20240501 07:58:38.493479 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:38.499320 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30f6292754d97a839700be93f8c07 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.499663 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30f6292754d97a839700be93f8c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.499620 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30f6292754d97a839700be93f8c07 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.499707 7120 tablet_service.cc:1467] Processing CreateTablet for tablet e11a47afed3a44afa75f0dd363098c22 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.499863 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30f6292754d97a839700be93f8c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.499940 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a47afed3a44afa75f0dd363098c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.501610 9134 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:38.501919 9135 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:38.502058 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30f6292754d97a839700be93f8c07 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.502331 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30f6292754d97a839700be93f8c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.502473 9134 tablet_bootstrap.cc:654] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.502698 9135 tablet_bootstrap.cc:654] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.502836 6989 tablet_service.cc:1467] Processing CreateTablet for tablet e11a47afed3a44afa75f0dd363098c22 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.503062 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a47afed3a44afa75f0dd363098c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.504012 9135 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:38.504101 9135 ts_tablet_manager.cc:1367] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.504510 9135 raft_consensus.cc:348] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.504631 9135 raft_consensus.cc:374] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.504662 9135 raft_consensus.cc:729] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.504787 9135 consensus_queue.cc:260] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.504771 7252 tablet_service.cc:1467] Processing CreateTablet for tablet e11a47afed3a44afa75f0dd363098c22 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.505036 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a47afed3a44afa75f0dd363098c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.505016 9138 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:38.505649 9135 ts_tablet_manager.cc:1398] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.505702 7250 tablet_service.cc:1467] Processing CreateTablet for tablet a9bd69f172f440a98debac09a97b1f8d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.505807 9135 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:38.505899 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bd69f172f440a98debac09a97b1f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.506551 9135 tablet_bootstrap.cc:654] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.507736 7120 tablet_service.cc:1467] Processing CreateTablet for tablet a9bd69f172f440a98debac09a97b1f8d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.507944 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bd69f172f440a98debac09a97b1f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.508008 9135 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:38.508126 9135 ts_tablet_manager.cc:1367] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.508510 9135 raft_consensus.cc:348] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.508591 9135 raft_consensus.cc:374] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.508617 9135 raft_consensus.cc:729] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.508721 9135 consensus_queue.cc:260] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.509003 9138 tablet_bootstrap.cc:654] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.509088 9135 ts_tablet_manager.cc:1398] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:38.509325 9140 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:38.510298 9140 tablet_bootstrap.cc:654] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.510804 9135 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:38.510779 9138 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:38.510761 6857 tablet_service.cc:1467] Processing CreateTablet for tablet a9bd69f172f440a98debac09a97b1f8d (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.510885 9138 ts_tablet_manager.cc:1367] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:38.511082 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bd69f172f440a98debac09a97b1f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.511721 9135 tablet_bootstrap.cc:654] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.511436 9138 raft_consensus.cc:348] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.512356 9138 raft_consensus.cc:374] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.512441 9138 raft_consensus.cc:729] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.512607 9138 consensus_queue.cc:260] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.513022 9138 ts_tablet_manager.cc:1398] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.513092 9135 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:38.513168 9135 ts_tablet_manager.cc:1367] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:38.513222 9138 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:38.513586 9135 raft_consensus.cc:348] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.513653 9135 raft_consensus.cc:374] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.513670 9135 raft_consensus.cc:729] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.513784 9134 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:38.513767 9135 consensus_queue.cc:260] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.513864 9134 ts_tablet_manager.cc:1367] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 07:58:38.514003 9135 ts_tablet_manager.cc:1398] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:38.514600 9140 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:38.514559 9134 raft_consensus.cc:348] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.514659 9134 raft_consensus.cc:374] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.514698 9134 raft_consensus.cc:729] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.514722 9140 ts_tablet_manager.cc:1367] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:38.514796 9134 consensus_queue.cc:260] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.515092 9134 ts_tablet_manager.cc:1398] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:38.515322 9140 raft_consensus.cc:348] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.515398 9140 raft_consensus.cc:374] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.515417 9140 raft_consensus.cc:729] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.515513 9140 consensus_queue.cc:260] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.516093 9140 ts_tablet_manager.cc:1398] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:38.517001 9140 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:38.517261 9134 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:38.517889 9140 tablet_bootstrap.cc:654] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.518178 9134 tablet_bootstrap.cc:654] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.519228 9140 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:38.519354 9140 ts_tablet_manager.cc:1367] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.519799 9140 raft_consensus.cc:348] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.519889 9140 raft_consensus.cc:374] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.519922 9140 raft_consensus.cc:729] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.520072 9140 consensus_queue.cc:260] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.520340 9140 ts_tablet_manager.cc:1398] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:38.520928 9134 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:38.521263 9134 ts_tablet_manager.cc:1367] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:38.521597 9134 raft_consensus.cc:348] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.521682 9134 raft_consensus.cc:374] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.521715 9134 raft_consensus.cc:729] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.521766 9138 tablet_bootstrap.cc:654] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.521875 9134 consensus_queue.cc:260] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.522179 9134 ts_tablet_manager.cc:1398] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:38.523741 9138 tablet_bootstrap.cc:492] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:38.523847 9138 ts_tablet_manager.cc:1367] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:38.524315 9138 raft_consensus.cc:348] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.524466 9138 raft_consensus.cc:374] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.524564 9138 raft_consensus.cc:729] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.524737 9138 consensus_queue.cc:260] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.525341 9138 ts_tablet_manager.cc:1398] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:38.556265 9061 raft_consensus.cc:482] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.556378 9120 raft_consensus.cc:482] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.556368 9061 raft_consensus.cc:504] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.556483 9120 raft_consensus.cc:504] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.556764 9061 leader_election.cc:288] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:38.556782 9120 leader_election.cc:288] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:38.557178 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a47afed3a44afa75f0dd363098c22" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:38.557260 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bd69f172f440a98debac09a97b1f8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:38.557307 7009 raft_consensus.cc:2454] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:38.557363 6878 raft_consensus.cc:2454] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:38.557518 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bd69f172f440a98debac09a97b1f8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:38.557546 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a47afed3a44afa75f0dd363098c22" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:38.557624 7139 raft_consensus.cc:2454] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:38.557642 7140 raft_consensus.cc:2454] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:38.557819 7208 leader_election.cc:302] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:38.557915 7206 leader_election.cc:302] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.558111 9120 raft_consensus.cc:2781] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:38.558164 9120 raft_consensus.cc:482] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:38.558198 9120 raft_consensus.cc:3037] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.558236 9061 raft_consensus.cc:2781] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:38.558281 9061 raft_consensus.cc:482] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:38.558318 9061 raft_consensus.cc:3037] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.559247 9120 raft_consensus.cc:504] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.559338 9061 raft_consensus.cc:504] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.559587 9120 leader_election.cc:288] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:38.559625 9061 leader_election.cc:288] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:38.559969 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a47afed3a44afa75f0dd363098c22" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:38.559952 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bd69f172f440a98debac09a97b1f8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:38.560065 7009 raft_consensus.cc:3037] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.560083 6877 raft_consensus.cc:3037] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.560947 7009 raft_consensus.cc:2454] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:38.560947 6877 raft_consensus.cc:2454] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:38.561118 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bd69f172f440a98debac09a97b1f8d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:38.561225 7140 raft_consensus.cc:3037] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.561254 7208 leader_election.cc:302] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:38.561396 9061 raft_consensus.cc:2781] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.561520 9061 raft_consensus.cc:686] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:38.561699 9061 consensus_queue.cc:237] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:38.562130 7140 raft_consensus.cc:2454] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:38.562306 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11a47afed3a44afa75f0dd363098c22" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:38.562390 7139 raft_consensus.cc:3037] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.562465 7206 leader_election.cc:302] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.562605 9061 raft_consensus.cc:2781] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.562803 9061 raft_consensus.cc:686] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:38.562919 9061 consensus_queue.cc:237] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.563186 7139 raft_consensus.cc:2454] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:38.564519 6735 catalog_manager.cc:5484] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:38.567678 6735 catalog_manager.cc:5484] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:38.683097 8866 raft_consensus.cc:482] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:38.683207 8866 raft_consensus.cc:504] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.683578 8866 leader_election.cc:288] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:38.683888 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c30f6292754d97a839700be93f8c07" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:38.683890 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c30f6292754d97a839700be93f8c07" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:38.684000 6877 raft_consensus.cc:2454] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:38.684007 7009 raft_consensus.cc:2454] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:38.684233 7076 leader_election.cc:302] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.684401 8866 raft_consensus.cc:2781] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:38.684454 8866 raft_consensus.cc:482] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:38.684478 8866 raft_consensus.cc:3037] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.685297 8866 raft_consensus.cc:504] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.685606 8866 leader_election.cc:288] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:38.685907 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c30f6292754d97a839700be93f8c07" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:38.685914 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c30f6292754d97a839700be93f8c07" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:38.686019 7009 raft_consensus.cc:3037] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.686017 6877 raft_consensus.cc:3037] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.686815 7009 raft_consensus.cc:2454] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:38.686967 6877 raft_consensus.cc:2454] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:38.687106 7077 leader_election.cc:302] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:38.687299 8866 raft_consensus.cc:2781] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.687472 8866 raft_consensus.cc:686] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:38.687599 8866 consensus_queue.cc:237] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:38.689438 6735 catalog_manager.cc:5484] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
W20240501 07:58:38.804251 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:38.804563 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:38.807803 9152 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 07:58:38.893422 7139 raft_consensus.cc:1261] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.893433 6878 raft_consensus.cc:1261] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.893772 6877 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.893990 9061 consensus_queue.cc:1035] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.894804 9148 consensus_queue.cc:1035] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.895331 7010 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.895326 9150 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.899199 7139 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.899696 9061 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.899817 7010 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:38.900173 8866 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:38.902752 9148 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.973 ms
time per row: 0.0534867 ms
I20240501 07:58:38.957437 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:46797
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:42471
--webserver_port=35241
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:58:39.073120 9189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.073341 9189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.073386 9189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.076668 9189 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:39.076759 9189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.076855 9189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:39.080633 9189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:46797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=35241
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.9189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.081526 9189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.082615 9189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.091912 9197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.092648 9195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.093516 9194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.094739 9189 server_base.cc:1034] running on GCE node
I20240501 07:58:39.095230 9189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.095849 9189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.097010 9189 hybrid_clock.cc:648] HybridClock initialized: now 1714550319096999 us; error 27 us; skew 500 ppm
I20240501 07:58:39.103113 9189 webserver.cc:460] Webserver started at http://127.0.99.133:35241/ using document root <none> and password file <none>
I20240501 07:58:39.103646 9189 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.103726 9189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.106954 9189 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:39.108729 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.109774 9189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:39.109877 9189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "49adb9e99822494196ed22044bfed81f"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:39.110294 9189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.125977 9189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.126744 9189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.126914 9189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.127521 9189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.128422 9189 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.128480 9189 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.128542 9189 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.128567 9189 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.138155 9189 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46797
I20240501 07:58:39.138178 9316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46797 every 8 connection(s)
I20240501 07:58:39.139205 9189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:39.143278 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 9189
I20240501 07:58:39.143672 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:42153
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:42471
--webserver_port=39949
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:39.148847 9317 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:39.149128 9317 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:39.149773 9317 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:39.151036 6735 ts_manager.cc:194] Re-registered known tserver with Master: 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:39.151824 6735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44131
W20240501 07:58:39.258939 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.259186 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.259243 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.262473 9320 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:39.262547 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.262631 9320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:39.266281 9320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:42153
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=39949
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.9320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.267385 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.268731 9320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.277758 9327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.278878 9326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.279886 9329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.280812 9320 server_base.cc:1034] running on GCE node
I20240501 07:58:39.281296 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.281931 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.283074 9320 hybrid_clock.cc:648] HybridClock initialized: now 1714550319283061 us; error 25 us; skew 500 ppm
I20240501 07:58:39.285113 9320 webserver.cc:460] Webserver started at http://127.0.99.134:39949/ using document root <none> and password file <none>
I20240501 07:58:39.285686 9320 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.285763 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.289451 9320 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 07:58:39.291219 9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.292215 9320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:39.292326 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fa6964452854415a85ad72f8639768d6"
format_stamp: "Formatted at 2024-05-01 07:58:23 on dist-test-slave-qm51"
I20240501 07:58:39.292742 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.311910 9320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.312536 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.312691 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.313198 9320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.314097 9320 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.314143 9320 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.314188 9320 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.314219 9320 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.323910 9320 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42153
I20240501 07:58:39.323968 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42153 every 8 connection(s)
I20240501 07:58:39.324993 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:39.329617 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 9320
I20240501 07:58:39.329998 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:36013
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:42471
--webserver_port=32771
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:39.334923 9449 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:39.335150 9449 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:39.335566 9449 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:39.336812 6735 ts_manager.cc:194] Re-registered known tserver with Master: fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:39.337653 6735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:59009
W20240501 07:58:39.444955 9452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.445166 9452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.445207 9452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.448346 9452 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:39.448412 9452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.448482 9452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:39.452087 9452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:36013
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=32771
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.9452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.453002 9452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.454075 9452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.462921 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.463660 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.464844 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.466055 9452 server_base.cc:1034] running on GCE node
I20240501 07:58:39.466470 9452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.467186 9452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.468348 9452 hybrid_clock.cc:648] HybridClock initialized: now 1714550319468320 us; error 46 us; skew 500 ppm
I20240501 07:58:39.470263 9452 webserver.cc:460] Webserver started at http://127.0.99.135:32771/ using document root <none> and password file <none>
I20240501 07:58:39.470795 9452 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.470862 9452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.474468 9452 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:39.476307 9467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.477329 9452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:39.477432 9452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ac6c8bc1b74b483791e262abee7862cf"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:39.477839 9452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.492352 9452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.492920 9452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.493043 9452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.493496 9452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.494292 9452 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.494335 9452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.494374 9452 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.494395 9452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.503454 9452 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:36013
I20240501 07:58:39.503508 9580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:36013 every 8 connection(s)
I20240501 07:58:39.504441 9452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:39.506268 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 9452
I20240501 07:58:39.506719 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:35111
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:42471
--webserver_port=41661
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:58:39.515321 9581 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:39.515601 9581 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:39.516186 9581 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:39.517252 6735 ts_manager.cc:194] Re-registered known tserver with Master: ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:39.518112 6735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:41539
W20240501 07:58:39.624531 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.624773 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.624821 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.628013 9584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:39.628093 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.628165 9584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:39.631892 9584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:35111
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=41661
--tserver_master_addrs=127.0.99.190:42471
--never_fsync=true
--heap_profile_path=/tmp/kudu.9584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.632892 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.634007 9584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.643711 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.644418 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.645171 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.646414 9584 server_base.cc:1034] running on GCE node
I20240501 07:58:39.646785 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.647490 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.648643 9584 hybrid_clock.cc:648] HybridClock initialized: now 1714550319648626 us; error 35 us; skew 500 ppm
I20240501 07:58:39.650921 9584 webserver.cc:460] Webserver started at http://127.0.99.136:41661/ using document root <none> and password file <none>
I20240501 07:58:39.651525 9584 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.651597 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.654956 9584 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:39.656915 9598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.657976 9584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:39.658085 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7fa3a88c25964445975bb47e0feed460"
format_stamp: "Formatted at 2024-05-01 07:58:24 on dist-test-slave-qm51"
I20240501 07:58:39.658514 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.683707 9584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.684372 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.684530 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.685081 9584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.686022 9584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.686080 9584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.686143 9584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.686184 9584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.695286 9584 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35111
I20240501 07:58:39.695339 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35111 every 8 connection(s)
I20240501 07:58:39.696460 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:39.703331 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 9584
I20240501 07:58:39.706563 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:39.707756 9712 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42471
I20240501 07:58:39.707986 9712 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:39.708478 9712 heartbeater.cc:511] Master 127.0.99.190:42471 requested a full tablet report, sending...
I20240501 07:58:39.709550 6737 ts_manager.cc:194] Re-registered known tserver with Master: 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:39.710211 6737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:33227
I20240501 07:58:39.714416 6857 tablet_service.cc:1467] Processing CreateTablet for tablet feb789ef04e74e8894c6688039dc3b1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]), partition=
I20240501 07:58:39.714720 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb789ef04e74e8894c6688039dc3b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.717716 9728 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:39.719985 9728 tablet_bootstrap.cc:654] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.720952 9728 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:39.721050 9728 ts_tablet_manager.cc:1367] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:39.721426 9728 raft_consensus.cc:348] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.721928 9728 raft_consensus.cc:374] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.722126 9728 raft_consensus.cc:729] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.722123 9383 tablet_service.cc:1467] Processing CreateTablet for tablet feb789ef04e74e8894c6688039dc3b1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]), partition=
I20240501 07:58:39.722291 9728 consensus_queue.cc:260] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.722654 9728 ts_tablet_manager.cc:1398] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:39.722826 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb789ef04e74e8894c6688039dc3b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.726418 9507 tablet_service.cc:1467] Processing CreateTablet for tablet feb789ef04e74e8894c6688039dc3b1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]), partition=
I20240501 07:58:39.727043 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb789ef04e74e8894c6688039dc3b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.730839 9730 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:39.733139 9731 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:39.732980 9730 tablet_bootstrap.cc:654] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.734046 9730 log.cc:824] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:39.734969 9731 tablet_bootstrap.cc:654] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.735633 9731 log.cc:824] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:39.735932 9730 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:39.736116 9730 ts_tablet_manager.cc:1367] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:58:39.736922 9731 tablet_bootstrap.cc:492] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:39.737064 9731 ts_tablet_manager.cc:1367] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:39.739734 9730 raft_consensus.cc:348] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.740000 9730 raft_consensus.cc:374] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.740049 9730 raft_consensus.cc:729] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.740375 9731 raft_consensus.cc:348] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.740564 9731 raft_consensus.cc:374] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.740511 9730 consensus_queue.cc:260] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.740633 9731 raft_consensus.cc:729] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.741039 9731 consensus_queue.cc:260] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.741430 9730 ts_tablet_manager.cc:1398] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240501 07:58:39.741498 9449 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
I20240501 07:58:39.742425 9731 ts_tablet_manager.cc:1398] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:58:39.742803 9581 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
W20240501 07:58:39.755290 9582 tablet.cc:2369] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:39.823369 9735 raft_consensus.cc:482] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:39.823702 9735 raft_consensus.cc:504] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.825896 9735 leader_election.cc:288] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
W20240501 07:58:39.826179 9450 tablet.cc:2369] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:39.829463 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb789ef04e74e8894c6688039dc3b1a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:39.829735 9403 raft_consensus.cc:2454] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:39.830164 9470 leader_election.cc:302] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:39.830437 9735 raft_consensus.cc:2781] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:39.830514 9735 raft_consensus.cc:482] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:39.830654 9735 raft_consensus.cc:3037] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.831609 9735 raft_consensus.cc:504] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.831655 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb789ef04e74e8894c6688039dc3b1a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:39.831804 6878 raft_consensus.cc:2454] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:39.832161 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb789ef04e74e8894c6688039dc3b1a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:39.832278 9403 raft_consensus.cc:3037] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.832283 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb789ef04e74e8894c6688039dc3b1a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:39.832379 6878 raft_consensus.cc:3037] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.833266 9403 raft_consensus.cc:2454] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:39.833266 6878 raft_consensus.cc:2454] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:39.831916 9735 leader_election.cc:288] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:39.833536 9470 leader_election.cc:302] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:39.833814 9735 raft_consensus.cc:2781] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.834012 9735 raft_consensus.cc:686] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:39.834337 9735 consensus_queue.cc:237] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:39.837320 6735 catalog_manager.cc:5484] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:39.962374 9238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:39.963487 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:39.968828 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:39.971086 9507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:39.976346 6878 raft_consensus.cc:1261] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:39.976830 9403 raft_consensus.cc:1261] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:39.977727 9741 consensus_queue.cc:1035] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:39.978145 9741 consensus_queue.cc:1035] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:39.978263 9646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:39.981520 7250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:39.984345 6734 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]). Expected version 1 got 0
I20240501 07:58:39.985386 6732 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]). Expected version 1 got 0
I20240501 07:58:39.988723 6989 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:58:39.993391 6732 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]). Expected version 1 got 0
I20240501 07:58:39.994529 6732 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903]). Expected version 1 got 0
I20240501 07:58:39.995894 9803 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:39.996785 9803 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:40.000161 7120 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:40.003485 9799 mvcc.cc:205] Tried to move back new op lower bound from 7022798110616440832 to 7022798110044667904. Current Snapshot: MvccSnapshot[applied={T|T < 7022798110616440832}]
I20240501 07:58:40.003779 9804 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:40.004415 9804 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:40.009166 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903] alter complete (version 1)
I20240501 07:58:40.011574 9797 mvcc.cc:205] Tried to move back new op lower bound from 7022798110616440832 to 7022798110044667904. Current Snapshot: MvccSnapshot[applied={T|T < 7022798110616440832}]
I20240501 07:58:40.033501 9802 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:40.034247 9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.035372 9802 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:40.116753 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:40.123320 9802 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.123644 9802 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:40.123791 9804 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.124094 9804 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:40.124114 9803 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.124359 9803 tablet.cc:1620] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:40.129150 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=9a89f286b7f94f269b8dfbc56f086903] alter complete (version 2)
I20240501 07:58:40.153863 9317 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
I20240501 07:58:40.181192 9238 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.182997 9646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.183503 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:40.183528 7250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:40.183766 6989 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:58:40.185213 7120 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:40.192559 9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.204612 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:40.206431 6731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 07:58:40.206564 6731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 07:58:40.209620 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 49bc04acac844b5ab0995fad69e51bbe
I20240501 07:58:40.210031 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 48b6f304fb9141fea296a48d35eb33ef
I20240501 07:58:40.210228 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 2a7033ccd32d43f49df11d96a8752276
I20240501 07:58:40.210909 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bc04acac844b5ab0995fad69e51bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:40.211300 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b6f304fb9141fea296a48d35eb33ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:40.211649 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b6f304fb9141fea296a48d35eb33ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:40.211975 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7033ccd32d43f49df11d96a8752276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:40.212288 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bc04acac844b5ab0995fad69e51bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:40.212358 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7033ccd32d43f49df11d96a8752276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:40.215296 9864 tablet_replica.cc:330] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:40.215209 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4ad48cbfb947f4aec462093006f6c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.215579 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4ad48cbfb947f4aec462093006f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.215878 9864 raft_consensus.cc:2229] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.216236 9864 raft_consensus.cc:2258] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.216451 9383 tablet_service.cc:1467] Processing CreateTablet for tablet c72ca77c85c94540905f236a4fc2e300 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.216825 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72ca77c85c94540905f236a4fc2e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.217300 9864 ts_tablet_manager.cc:1875] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.219111 9730 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:40.219599 9864 ts_tablet_manager.cc:1888] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.219681 9864 log.cc:1196] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/49bc04acac844b5ab0995fad69e51bbe
I20240501 07:58:40.219985 9864 ts_tablet_manager.cc:1909] T 49bc04acac844b5ab0995fad69e51bbe P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:40.220947 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 72937d60b2014cb490cbbb89c1082519 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.221186 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72937d60b2014cb490cbbb89c1082519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.221238 9864 tablet_replica.cc:330] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:40.221360 9864 raft_consensus.cc:2229] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.221589 9864 raft_consensus.cc:2258] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.222076 9864 ts_tablet_manager.cc:1875] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.222347 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 49bc04acac844b5ab0995fad69e51bbe (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.223721 9731 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:40.223852 9864 ts_tablet_manager.cc:1888] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.223912 9864 log.cc:1196] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/48b6f304fb9141fea296a48d35eb33ef
I20240501 07:58:40.224145 9864 ts_tablet_manager.cc:1909] T 48b6f304fb9141fea296a48d35eb33ef P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:40.224401 9866 tablet_replica.cc:330] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:40.224570 9731 tablet_bootstrap.cc:654] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.224676 9866 raft_consensus.cc:2229] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.224972 9866 raft_consensus.cc:2258] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.224785 9645 tablet_service.cc:1467] Processing CreateTablet for tablet c72ca77c85c94540905f236a4fc2e300 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.225593 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72ca77c85c94540905f236a4fc2e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.225741 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b6f304fb9141fea296a48d35eb33ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.225869 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7033ccd32d43f49df11d96a8752276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.225772 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bc04acac844b5ab0995fad69e51bbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.226516 9866 ts_tablet_manager.cc:1875] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.227416 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4ad48cbfb947f4aec462093006f6c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.224665 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 72937d60b2014cb490cbbb89c1082519 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.228229 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4ad48cbfb947f4aec462093006f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.228369 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72937d60b2014cb490cbbb89c1082519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.230940 9866 ts_tablet_manager.cc:1888] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.231099 9866 log.cc:1196] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/49bc04acac844b5ab0995fad69e51bbe
I20240501 07:58:40.231267 9251 tablet_service.cc:1467] Processing CreateTablet for tablet c72ca77c85c94540905f236a4fc2e300 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.231438 9866 ts_tablet_manager.cc:1909] T 49bc04acac844b5ab0995fad69e51bbe P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:40.231506 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72ca77c85c94540905f236a4fc2e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.232375 9866 tablet_replica.cc:330] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:40.232488 9866 raft_consensus.cc:2229] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.232638 9866 raft_consensus.cc:2258] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.233065 9866 ts_tablet_manager.cc:1875] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.224664 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4ad48cbfb947f4aec462093006f6c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.234426 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4ad48cbfb947f4aec462093006f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.234900 9866 ts_tablet_manager.cc:1888] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.234954 9866 log.cc:1196] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a7033ccd32d43f49df11d96a8752276
I20240501 07:58:40.235200 9866 ts_tablet_manager.cc:1909] T 2a7033ccd32d43f49df11d96a8752276 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:40.235230 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 72937d60b2014cb490cbbb89c1082519 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:40.235456 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72937d60b2014cb490cbbb89c1082519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.237464 9730 tablet_bootstrap.cc:654] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.238410 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 48b6f304fb9141fea296a48d35eb33ef (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.239032 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 49bc04acac844b5ab0995fad69e51bbe (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.239166 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 2a7033ccd32d43f49df11d96a8752276 (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.239140 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:40.242545 9730 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:40.242641 9730 ts_tablet_manager.cc:1367] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240501 07:58:40.243513 9730 raft_consensus.cc:348] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.243636 9730 raft_consensus.cc:374] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.243678 9730 raft_consensus.cc:729] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.243799 9730 consensus_queue.cc:260] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.244050 9730 ts_tablet_manager.cc:1398] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:40.244199 9730 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:40.244959 9730 tablet_bootstrap.cc:654] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.246342 9802 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.248994 9804 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.251821 9803 tablet.cc:1722] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.254443 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=9a89f286b7f94f269b8dfbc56f086903] alter complete (version 3)
I20240501 07:58:40.254575 9870 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:40.257120 9507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:40.257594 9730 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:40.257678 9730 ts_tablet_manager.cc:1367] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:58:40.258139 9730 raft_consensus.cc:348] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.258220 9730 raft_consensus.cc:374] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.258240 9730 raft_consensus.cc:729] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.258348 9730 consensus_queue.cc:260] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.258646 9730 ts_tablet_manager.cc:1398] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:40.263188 9865 tablet_replica.cc:330] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:40.263515 9865 raft_consensus.cc:2229] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.263737 9865 raft_consensus.cc:2258] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.265347 9865 ts_tablet_manager.cc:1875] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.267084 9869 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:40.267670 9868 tablet_replica.cc:330] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.268041 9865 ts_tablet_manager.cc:1888] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.268062 9868 raft_consensus.cc:2229] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.268186 9865 log.cc:1196] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/48b6f304fb9141fea296a48d35eb33ef
I20240501 07:58:40.268481 9870 tablet_bootstrap.cc:654] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.268587 9865 ts_tablet_manager.cc:1909] T 48b6f304fb9141fea296a48d35eb33ef P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:40.269160 9869 tablet_bootstrap.cc:654] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.269371 9868 raft_consensus.cc:2258] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.269860 9869 log.cc:824] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:40.269986 9865 tablet_replica.cc:330] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:40.270380 9868 ts_tablet_manager.cc:1875] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.270588 9865 raft_consensus.cc:2229] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.270792 9865 raft_consensus.cc:2258] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.270865 9731 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:40.270954 9731 ts_tablet_manager.cc:1367] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240501 07:58:40.271485 9731 raft_consensus.cc:348] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.271603 9731 raft_consensus.cc:374] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.271639 9731 raft_consensus.cc:729] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.271757 9731 consensus_queue.cc:260] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.272008 9731 ts_tablet_manager.cc:1398] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:40.271431 9865 ts_tablet_manager.cc:1875] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.272763 9868 ts_tablet_manager.cc:1888] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.273092 9870 log.cc:824] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:40.273300 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 48b6f304fb9141fea296a48d35eb33ef (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.276031 9865 ts_tablet_manager.cc:1888] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.276161 9865 log.cc:1196] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2a7033ccd32d43f49df11d96a8752276
I20240501 07:58:40.276463 9865 ts_tablet_manager.cc:1909] T 2a7033ccd32d43f49df11d96a8752276 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:40.277706 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 2a7033ccd32d43f49df11d96a8752276 (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.278764 9869 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:40.278918 9870 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:40.279027 9869 ts_tablet_manager.cc:1367] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240501 07:58:40.279191 9870 ts_tablet_manager.cc:1367] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240501 07:58:40.282941 9870 raft_consensus.cc:348] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.283326 9868 log.cc:1196] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/48b6f304fb9141fea296a48d35eb33ef
I20240501 07:58:40.284406 9869 raft_consensus.cc:348] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.284596 9869 raft_consensus.cc:374] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.284651 9869 raft_consensus.cc:729] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.285092 9734 raft_consensus.cc:482] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:40.285202 9734 raft_consensus.cc:504] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.285276 9868 ts_tablet_manager.cc:1909] T 48b6f304fb9141fea296a48d35eb33ef P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.286311 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 48b6f304fb9141fea296a48d35eb33ef (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.286346 9734 leader_election.cc:288] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:40.288156 9869 consensus_queue.cc:260] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.288693 9868 tablet_replica.cc:330] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.288866 9741 raft_consensus.cc:482] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:40.288914 9868 raft_consensus.cc:2229] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.288919 9712 heartbeater.cc:503] Master 127.0.99.190:42471 was elected leader, sending a full tablet report...
I20240501 07:58:40.288960 9741 raft_consensus.cc:504] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.289064 9868 raft_consensus.cc:2258] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.289635 9869 ts_tablet_manager.cc:1398] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20240501 07:58:40.289645 9868 ts_tablet_manager.cc:1875] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.289899 9869 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:40.291185 9869 tablet_bootstrap.cc:654] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.291975 9868 ts_tablet_manager.cc:1888] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.292058 9868 log.cc:1196] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/49bc04acac844b5ab0995fad69e51bbe
I20240501 07:58:40.292169 9870 raft_consensus.cc:374] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.292240 9870 raft_consensus.cc:729] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.292392 9869 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:40.292466 9869 ts_tablet_manager.cc:1367] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:40.292727 9870 consensus_queue.cc:260] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.292985 9869 raft_consensus.cc:348] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.293078 9869 raft_consensus.cc:374] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.293107 9869 raft_consensus.cc:729] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.293210 9869 consensus_queue.cc:260] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.293618 9870 ts_tablet_manager.cc:1398] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240501 07:58:40.294092 9868 ts_tablet_manager.cc:1909] T 49bc04acac844b5ab0995fad69e51bbe P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.294353 9870 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:40.294593 9869 ts_tablet_manager.cc:1398] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:40.294780 9869 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:40.295302 9868 tablet_replica.cc:330] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.295655 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 49bc04acac844b5ab0995fad69e51bbe (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
I20240501 07:58:40.296159 9868 raft_consensus.cc:2229] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.296550 9870 tablet_bootstrap.cc:654] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.297035 9868 raft_consensus.cc:2258] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.297611 9868 ts_tablet_manager.cc:1875] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.298594 9869 tablet_bootstrap.cc:654] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.298940 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:40.299232 9741 leader_election.cc:288] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:40.299314 9266 raft_consensus.cc:2454] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:40.299927 9338 leader_election.cc:302] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:40.300431 9734 raft_consensus.cc:2781] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:40.300498 9734 raft_consensus.cc:482] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:40.300540 9734 raft_consensus.cc:3037] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.301488 9868 ts_tablet_manager.cc:1888] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.301545 9868 log.cc:1196] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a7033ccd32d43f49df11d96a8752276
I20240501 07:58:40.301479 9734 raft_consensus.cc:504] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.301796 9868 ts_tablet_manager.cc:1909] T 2a7033ccd32d43f49df11d96a8752276 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.301904 9734 leader_election.cc:288] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:40.302505 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:40.302620 9266 raft_consensus.cc:3037] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.303720 9266 raft_consensus.cc:2454] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:40.304029 9338 leader_election.cc:302] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:40.304226 9734 raft_consensus.cc:2781] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:40.304409 9734 raft_consensus.cc:686] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:40.304714 9734 consensus_queue.cc:237] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.304982 9870 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:40.305114 9870 ts_tablet_manager.cc:1367] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:40.305702 9870 raft_consensus.cc:348] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.305882 9870 raft_consensus.cc:374] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.305972 9870 raft_consensus.cc:729] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.306165 9870 consensus_queue.cc:260] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.306552 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:40.306597 9870 ts_tablet_manager.cc:1398] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:40.306775 9870 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:40.307210 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 2a7033ccd32d43f49df11d96a8752276 (table rebalance_test_table_0 [id=fa9c73603129441386c84812dabd23b9]) successfully deleted
W20240501 07:58:40.307186 9470 leader_election.cc:341] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:40.307924 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:40.308068 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
W20240501 07:58:40.308598 9338 leader_election.cc:341] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Illegal state: must be running to vote when last-logged opid is not known
W20240501 07:58:40.308934 9338 leader_election.cc:341] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:40.308938 6733 catalog_manager.cc:5484] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.309181 9870 tablet_bootstrap.cc:654] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.310355 9870 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:40.310446 9870 ts_tablet_manager.cc:1367] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:40.310859 9870 raft_consensus.cc:348] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.310963 9870 raft_consensus.cc:374] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.310997 9870 raft_consensus.cc:729] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.311174 9870 consensus_queue.cc:260] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.311348 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:40.311444 9870 ts_tablet_manager.cc:1398] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:40.311575 9665 raft_consensus.cc:2454] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:40.312058 9470 leader_election.cc:302] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters: 49adb9e99822494196ed22044bfed81f
I20240501 07:58:40.312284 9869 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:40.312322 9741 raft_consensus.cc:2781] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:40.312381 9741 raft_consensus.cc:482] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:40.312387 9869 ts_tablet_manager.cc:1367] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:58:40.312427 9741 raft_consensus.cc:3037] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.313162 9869 raft_consensus.cc:348] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.313280 9869 raft_consensus.cc:374] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.313315 9869 raft_consensus.cc:729] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.313334 9741 raft_consensus.cc:504] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.313436 9869 consensus_queue.cc:260] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.313716 9869 ts_tablet_manager.cc:1398] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:40.313716 9741 leader_election.cc:288] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:40.314064 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:40.314170 9665 raft_consensus.cc:3037] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.314426 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:40.314538 9266 raft_consensus.cc:3037] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.315150 9665 raft_consensus.cc:2454] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:40.315439 9470 leader_election.cc:302] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:40.315611 9266 raft_consensus.cc:2454] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:40.315646 9741 raft_consensus.cc:2781] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:40.315735 9741 raft_consensus.cc:686] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:40.315856 9741 consensus_queue.cc:237] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.317363 6735 catalog_manager.cc:5484] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.356490 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:40.356635 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:40.358795 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet feb789ef04e74e8894c6688039dc3b1a
I20240501 07:58:40.359417 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet feb789ef04e74e8894c6688039dc3b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:40.359719 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet feb789ef04e74e8894c6688039dc3b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:40.359891 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet feb789ef04e74e8894c6688039dc3b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.359932 9903 tablet_replica.cc:330] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:40.360077 9868 tablet_replica.cc:330] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.360198 9868 raft_consensus.cc:2229] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.360227 9903 raft_consensus.cc:2229] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.360322 9868 raft_consensus.cc:2258] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.360379 9904 tablet_replica.cc:330] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:40.360630 9903 raft_consensus.cc:2258] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.360699 9904 raft_consensus.cc:2229] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.360939 9904 raft_consensus.cc:2258] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.361020 9868 ts_tablet_manager.cc:1875] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.362982 9868 ts_tablet_manager.cc:1888] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:40.363099 9868 log.cc:1196] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/feb789ef04e74e8894c6688039dc3b1a
I20240501 07:58:40.363379 9868 ts_tablet_manager.cc:1909] T feb789ef04e74e8894c6688039dc3b1a P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.363709 9903 ts_tablet_manager.cc:1875] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.363983 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table_new_name [id=9a89f286b7f94f269b8dfbc56f086903]) successfully deleted
I20240501 07:58:40.364460 9904 ts_tablet_manager.cc:1875] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.365535 9903 ts_tablet_manager.cc:1888] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:40.365607 9903 log.cc:1196] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/feb789ef04e74e8894c6688039dc3b1a
I20240501 07:58:40.365868 9903 ts_tablet_manager.cc:1909] T feb789ef04e74e8894c6688039dc3b1a P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:40.366283 9904 ts_tablet_manager.cc:1888] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:40.366375 9904 log.cc:1196] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/feb789ef04e74e8894c6688039dc3b1a
I20240501 07:58:40.366717 9904 ts_tablet_manager.cc:1909] T feb789ef04e74e8894c6688039dc3b1a P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:40.367077 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table_new_name [id=9a89f286b7f94f269b8dfbc56f086903]) successfully deleted
I20240501 07:58:40.367774 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet feb789ef04e74e8894c6688039dc3b1a (table rebalancer_dynamic_table_new_name [id=9a89f286b7f94f269b8dfbc56f086903]) successfully deleted
W20240501 07:58:40.391263 9318 tablet.cc:2369] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:40.447767 9713 tablet.cc:2369] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:40.451529 9877 raft_consensus.cc:482] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:40.451656 9877 raft_consensus.cc:504] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.452741 9877 leader_election.cc:288] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:40.456629 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:40.456848 9266 raft_consensus.cc:2379] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fa3a88c25964445975bb47e0feed460 in current term 1: Already voted for candidate fa6964452854415a85ad72f8639768d6 in this term.
I20240501 07:58:40.457022 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72ca77c85c94540905f236a4fc2e300" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:40.457361 9601 leader_election.cc:302] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6
I20240501 07:58:40.457597 9877 raft_consensus.cc:3037] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.458451 9877 raft_consensus.cc:2726] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:40.461047 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:40.466034 9238 tablet_service.cc:1467] Processing CreateTablet for tablet a071981a53d741ce84778a3a6f7b3a01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]), partition=
I20240501 07:58:40.466290 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071981a53d741ce84778a3a6f7b3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.467754 9382 tablet_service.cc:1467] Processing CreateTablet for tablet a071981a53d741ce84778a3a6f7b3a01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]), partition=
I20240501 07:58:40.468015 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071981a53d741ce84778a3a6f7b3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.468350 9870 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:40.468497 6990 tablet_service.cc:1467] Processing CreateTablet for tablet a071981a53d741ce84778a3a6f7b3a01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]), partition=
I20240501 07:58:40.468715 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a071981a53d741ce84778a3a6f7b3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:40.469646 9870 tablet_bootstrap.cc:654] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.470278 9730 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:40.471138 9870 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:40.471225 9870 ts_tablet_manager.cc:1367] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:40.471305 9730 tablet_bootstrap.cc:654] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.471710 9870 raft_consensus.cc:348] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.471824 9870 raft_consensus.cc:374] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.471864 9870 raft_consensus.cc:729] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.471990 9870 consensus_queue.cc:260] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.472954 9870 ts_tablet_manager.cc:1398] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:40.473475 9730 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:40.473568 9730 ts_tablet_manager.cc:1367] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:40.474013 9730 raft_consensus.cc:348] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.474086 9730 raft_consensus.cc:374] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.474107 9730 raft_consensus.cc:729] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.474208 9730 consensus_queue.cc:260] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.474449 9730 ts_tablet_manager.cc:1398] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:40.474730 9934 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:40.475598 9934 tablet_bootstrap.cc:654] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:40.476620 9934 tablet_bootstrap.cc:492] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:40.476716 9934 ts_tablet_manager.cc:1367] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:40.477066 9934 raft_consensus.cc:348] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.477161 9934 raft_consensus.cc:374] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:40.477193 9934 raft_consensus.cc:729] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.477304 9934 consensus_queue.cc:260] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.477567 9934 ts_tablet_manager.cc:1398] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:40.545468 9879 raft_consensus.cc:482] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:40.545622 9879 raft_consensus.cc:504] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.546706 9879 leader_election.cc:288] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:40.552057 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72937d60b2014cb490cbbb89c1082519" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:40.552199 9403 raft_consensus.cc:2454] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:40.552518 9206 leader_election.cc:302] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:40.552752 9879 raft_consensus.cc:2781] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:40.552824 9879 raft_consensus.cc:482] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:40.552881 9879 raft_consensus.cc:3037] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.553082 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72937d60b2014cb490cbbb89c1082519" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:40.553211 9665 raft_consensus.cc:2454] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:40.554006 9879 raft_consensus.cc:504] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.554347 9879 leader_election.cc:288] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:40.554685 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72937d60b2014cb490cbbb89c1082519" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:40.554714 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72937d60b2014cb490cbbb89c1082519" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:40.554795 9403 raft_consensus.cc:3037] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.554816 9665 raft_consensus.cc:3037] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.555686 9665 raft_consensus.cc:2454] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:40.555768 9403 raft_consensus.cc:2454] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:40.555970 9206 leader_election.cc:302] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:40.556107 9879 raft_consensus.cc:2781] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:40.556304 9879 raft_consensus.cc:686] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:40.556627 9879 consensus_queue.cc:237] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.559553 6737 catalog_manager.cc:5484] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.589744 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:40.593662 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.594199 6990 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:40.594669 6858 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:40.596145 9882 raft_consensus.cc:482] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:40.596246 9882 raft_consensus.cc:504] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.596799 9882 leader_election.cc:288] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:40.597589 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071981a53d741ce84778a3a6f7b3a01" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:40.597756 9266 raft_consensus.cc:2454] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:40.598124 9338 leader_election.cc:302] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:40.598526 9882 raft_consensus.cc:2781] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:40.598613 9882 raft_consensus.cc:482] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:40.598654 9882 raft_consensus.cc:3037] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.600448 9882 raft_consensus.cc:504] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.600780 9882 leader_election.cc:288] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:40.601226 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071981a53d741ce84778a3a6f7b3a01" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:40.601347 9266 raft_consensus.cc:3037] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.601496 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071981a53d741ce84778a3a6f7b3a01" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:40.601619 7009 raft_consensus.cc:2454] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:40.602246 9266 raft_consensus.cc:2454] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:40.602473 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a071981a53d741ce84778a3a6f7b3a01" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:40.602578 7010 raft_consensus.cc:3037] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:40.603498 7010 raft_consensus.cc:2454] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:40.603801 9339 leader_election.cc:302] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:40.604041 9882 raft_consensus.cc:2781] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:40.604117 9882 raft_consensus.cc:686] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:40.604250 9882 consensus_queue.cc:237] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.605496 9238 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:40.606083 6730 catalog_manager.cc:5484] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.620445 7121 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:40.627628 9507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:40.628445 9382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:40.701516 7141 consensus_queue.cc:237] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:40.703106 6878 raft_consensus.cc:1261] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.703472 7010 raft_consensus.cc:1261] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.703522 9150 consensus_queue.cc:1035] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.704159 9150 consensus_queue.cc:1035] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.705987 9942 raft_consensus.cc:2932] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.709115 6730 catalog_manager.cc:5484] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.709939 6878 raft_consensus.cc:2932] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.710759 7010 raft_consensus.cc:2932] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
W20240501 07:58:40.713249 7077 consensus_peers.cc:485] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e05498632f0945c29f723a4613a26a04. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.718012 6731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 07:58:40.718175 6731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 07:58:40.720645 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 58dc740b379142029e23f32382abc7b3
I20240501 07:58:40.720928 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 2cf7cfe95b0c44f480f4bf2dd8c78769
I20240501 07:58:40.721170 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 017dd2787fba49b7aca26fe954d8fab4
I20240501 07:58:40.721194 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 58dc740b379142029e23f32382abc7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.721278 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 58dc740b379142029e23f32382abc7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:40.721457 9866 tablet_replica.cc:330] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:40.721496 9868 tablet_replica.cc:330] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.721534 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 58dc740b379142029e23f32382abc7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:40.721586 9866 raft_consensus.cc:2229] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.721632 9868 raft_consensus.cc:2229] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.721753 9868 raft_consensus.cc:2258] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.721776 9866 raft_consensus.cc:2258] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.722132 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:40.722254 9866 ts_tablet_manager.cc:1875] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.722254 9868 ts_tablet_manager.cc:1875] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.724387 9868 ts_tablet_manager.cc:1888] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.724387 9866 ts_tablet_manager.cc:1888] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.724452 9868 log.cc:1196] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/58dc740b379142029e23f32382abc7b3
I20240501 07:58:40.724452 9866 log.cc:1196] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/58dc740b379142029e23f32382abc7b3
I20240501 07:58:40.724723 9868 ts_tablet_manager.cc:1909] T 58dc740b379142029e23f32382abc7b3 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.724737 9866 ts_tablet_manager.cc:1909] T 58dc740b379142029e23f32382abc7b3 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:40.724961 9864 tablet_replica.cc:330] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:40.725111 9864 raft_consensus.cc:2229] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.725265 9864 raft_consensus.cc:2258] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.725396 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 58dc740b379142029e23f32382abc7b3 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.725529 9866 tablet_replica.cc:330] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:40.725627 9866 raft_consensus.cc:2229] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.725719 9866 raft_consensus.cc:2258] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.725718 9864 ts_tablet_manager.cc:1875] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.726078 9866 ts_tablet_manager.cc:1875] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.727365 9864 ts_tablet_manager.cc:1888] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.727377 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:40.727434 9864 log.cc:1196] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/58dc740b379142029e23f32382abc7b3
I20240501 07:58:40.727416 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 58dc740b379142029e23f32382abc7b3 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.727571 9865 tablet_replica.cc:330] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:40.727686 9864 ts_tablet_manager.cc:1909] T 58dc740b379142029e23f32382abc7b3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:40.727727 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:40.727718 9865 raft_consensus.cc:2229] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.727876 9865 raft_consensus.cc:2258] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.728011 9866 ts_tablet_manager.cc:1888] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.728087 9866 log.cc:1196] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/017dd2787fba49b7aca26fe954d8fab4
I20240501 07:58:40.728335 9866 ts_tablet_manager.cc:1909] T 017dd2787fba49b7aca26fe954d8fab4 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:40.728335 9865 ts_tablet_manager.cc:1875] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.728607 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 58dc740b379142029e23f32382abc7b3 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.728621 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:40.728757 9864 tablet_replica.cc:330] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:40.728865 9864 raft_consensus.cc:2229] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.728912 9866 tablet_replica.cc:330] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:40.729018 9864 raft_consensus.cc:2258] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.729025 9866 raft_consensus.cc:2229] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.729132 9866 raft_consensus.cc:2258] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.729468 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 017dd2787fba49b7aca26fe954d8fab4 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.729493 9864 ts_tablet_manager.cc:1875] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.729835 9866 ts_tablet_manager.cc:1875] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.730405 9865 ts_tablet_manager.cc:1888] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.730470 9865 log.cc:1196] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2cf7cfe95b0c44f480f4bf2dd8c78769
I20240501 07:58:40.730736 9865 ts_tablet_manager.cc:1909] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:40.730955 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 017dd2787fba49b7aca26fe954d8fab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.731154 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:40 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:40.731181 9868 tablet_replica.cc:330] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.731312 9868 raft_consensus.cc:2229] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:40.731438 9868 raft_consensus.cc:2258] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.731854 9866 ts_tablet_manager.cc:1888] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.731966 9866 log.cc:1196] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2cf7cfe95b0c44f480f4bf2dd8c78769
I20240501 07:58:40.732220 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.732254 9866 ts_tablet_manager.cc:1909] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:40.732827 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.733309 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:40.733680 9868 ts_tablet_manager.cc:1875] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.734081 9864 ts_tablet_manager.cc:1888] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.734153 9864 log.cc:1196] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/017dd2787fba49b7aca26fe954d8fab4
I20240501 07:58:40.734443 9864 ts_tablet_manager.cc:1909] T 017dd2787fba49b7aca26fe954d8fab4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:40.735157 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 017dd2787fba49b7aca26fe954d8fab4 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.736598 9868 ts_tablet_manager.cc:1888] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.736670 9868 log.cc:1196] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/017dd2787fba49b7aca26fe954d8fab4
I20240501 07:58:40.736922 9868 ts_tablet_manager.cc:1909] T 017dd2787fba49b7aca26fe954d8fab4 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.737529 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 017dd2787fba49b7aca26fe954d8fab4 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.737553 9868 tablet_replica.cc:330] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:40.737685 9868 raft_consensus.cc:2229] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:40.737946 9868 raft_consensus.cc:2258] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:40.738448 9868 ts_tablet_manager.cc:1875] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:40.738891 9266 raft_consensus.cc:1261] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:40.739477 9734 consensus_queue.cc:1035] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.739580 7010 raft_consensus.cc:1261] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:40.740104 9734 consensus_queue.cc:1035] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.740296 9868 ts_tablet_manager.cc:1888] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:40.740368 9868 log.cc:1196] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2cf7cfe95b0c44f480f4bf2dd8c78769
I20240501 07:58:40.740607 9868 ts_tablet_manager.cc:1909] T 2cf7cfe95b0c44f480f4bf2dd8c78769 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:40.741153 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 2cf7cfe95b0c44f480f4bf2dd8c78769 (table rebalance_test_table_2 [id=fbc94645af2547f2897d473147120dc0]) successfully deleted
I20240501 07:58:40.742403 7010 consensus_queue.cc:237] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.743310 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]). Expected version 1 got 0
I20240501 07:58:40.744112 9804 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:40.744427 9804 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:40.744578 6878 raft_consensus.cc:1261] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.744988 9324 consensus_queue.cc:1035] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.746255 6737 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]). Expected version 1 got 0
I20240501 07:58:40.747509 7270 raft_consensus.cc:1261] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.747694 9998 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:40.748433 9998 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:40.748432 9936 consensus_queue.cc:1035] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.748579 9991 mvcc.cc:205] Tried to move back new op lower bound from 7022798113739460608 to 7022798113196552192. Current Snapshot: MvccSnapshot[applied={T|T < 7022798113739460608}]
I20240501 07:58:40.749442 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]). Expected version 1 got 0
I20240501 07:58:40.749744 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849] alter complete (version 1)
I20240501 07:58:40.753047 6737 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849]). Expected version 1 got 0
I20240501 07:58:40.754228 10000 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240501 07:58:40.754505 6945 consensus_peers.cc:485] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de03fdd63bd347dea4eb6e5d410da68e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.754746 9324 raft_consensus.cc:2932] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.754997 10000 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:40.756515 6878 raft_consensus.cc:2932] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.756862 7270 raft_consensus.cc:2932] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.759568 6731 catalog_manager.cc:5484] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.776782 7009 consensus_queue.cc:237] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.778076 7141 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.778295 6878 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.778483 9932 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.778705 9936 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.781827 6945 consensus_peers.cc:485] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58aab13671d647e485f9be94283b5239. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.790604 9936 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.791044 6878 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.794781 6737 catalog_manager.cc:5484] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.794883 7141 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.806488 9741 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.806949 9741 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.817638 7270 consensus_queue.cc:237] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.818965 6878 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.820021 9796 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.820052 7139 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.820385 9796 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.822678 7207 consensus_peers.cc:485] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1671c966902c4eb98b238c4419c525fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.822619 10041 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.822948 7139 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.824323 9882 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.825840 6730 catalog_manager.cc:5484] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.828352 9882 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:40.829294 6878 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.848150 7009 consensus_queue.cc:237] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:40.849478 6878 raft_consensus.cc:1261] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.849946 7270 raft_consensus.cc:1261] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.850091 9932 consensus_queue.cc:1035] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.850395 9932 consensus_queue.cc:1035] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.852110 9324 raft_consensus.cc:2932] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.852816 7271 raft_consensus.cc:2932] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.852994 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:40.855748 6730 catalog_manager.cc:5484] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.855937 6878 raft_consensus.cc:2932] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
W20240501 07:58:40.859614 6946 consensus_peers.cc:485] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33e711b7a9684137969ce2bbffae40c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.861284 9804 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.861654 9804 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:40.862636 9998 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.862970 9998 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:40.864198 10000 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:40.864495 10000 tablet.cc:1620] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:40.866343 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=0370673905e24aedaaa54798f6e7a849] alter complete (version 2)
I20240501 07:58:40.881922 7010 consensus_queue.cc:237] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:40.883352 7271 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.883574 6878 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.883832 10020 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.884020 6945 consensus_peers.cc:485] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d3d410c640243848a478aa2605c1bf0. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.884224 10020 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.885987 10020 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.886314 6877 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.888409 7271 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.888883 6731 catalog_manager.cc:5484] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:40.909461 6877 consensus_queue.cc:237] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.910933 7010 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.911413 7139 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.911715 10100 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.912020 9863 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.914152 10100 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.914593 7139 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
W20240501 07:58:40.915665 6814 consensus_peers.cc:485] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4882718ff90c4d939917f4970a9acb9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.916908 7010 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.917436 6737 catalog_manager.cc:5484] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:40.937191 7139 consensus_queue.cc:237] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:40.938853 6877 raft_consensus.cc:1261] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.938912 7010 raft_consensus.cc:1261] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.939402 9972 consensus_queue.cc:1035] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.939620 9971 consensus_queue.cc:1035] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.942277 7076 consensus_peers.cc:485] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f5dd2edfd7b4e28b226299dc2a4fc04. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.942348 9972 raft_consensus.cc:2932] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.942767 6877 raft_consensus.cc:2932] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.944355 7010 raft_consensus.cc:2932] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.947468 6729 catalog_manager.cc:5484] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:40.966823 7139 consensus_queue.cc:237] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:40.968086 7271 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.968279 7010 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:40.968478 7077 consensus_peers.cc:485] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c80723cb0ce4446483a5f1754e066081. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.968685 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:40.968713 9942 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.968899 9971 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.971347 9971 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.972677 7271 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.975523 6729 catalog_manager.cc:5484] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.977246 7010 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:40.978046 9804 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.980823 9998 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.982965 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=0370673905e24aedaaa54798f6e7a849] alter complete (version 3)
I20240501 07:58:40.983345 10000 tablet.cc:1722] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:40.996245 7139 consensus_queue.cc:237] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:40.998152 7271 raft_consensus.cc:1261] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.998220 6877 raft_consensus.cc:1261] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:40.998538 9972 consensus_queue.cc:1035] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.998708 9942 consensus_queue.cc:1035] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.000622 9972 raft_consensus.cc:2932] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.001075 7271 raft_consensus.cc:2932] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
W20240501 07:58:41.001680 7076 consensus_peers.cc:485] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37e73deb03274bba82ceb67087208bdd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.002418 6877 raft_consensus.cc:2932] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.004544 6737 catalog_manager.cc:5484] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.022064 6877 consensus_queue.cc:237] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.023701 7010 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.023733 7139 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.024152 10001 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.024307 10100 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:41.027475 6814 consensus_peers.cc:485] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43b82e035ecb4e23973a2163e15c1dac. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.027746 10100 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.028187 7139 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.030717 6730 catalog_manager.cc:5484] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.031203 7009 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.053916 7271 consensus_queue.cc:237] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.055466 7139 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.056123 10041 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.056241 6877 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.056818 10041 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.058722 10041 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.059320 6877 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.060508 7139 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
W20240501 07:58:41.061558 7207 consensus_peers.cc:485] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7dcdd38eff43fe9fff9081ee644c8d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.062711 6729 catalog_manager.cc:5484] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.082062 7139 consensus_queue.cc:237] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.083369 7271 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:41.083686 7076 consensus_peers.cc:485] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00cac44dfd0f4d97a3056b590748ba58. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.083683 7009 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.083792 9972 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.084105 9971 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.085893 9972 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.086256 7009 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.087512 7271 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.087978 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:41.088099 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:41.089720 6737 catalog_manager.cc:5484] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.091152 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet a071981a53d741ce84778a3a6f7b3a01
I20240501 07:58:41.092270 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet a071981a53d741ce84778a3a6f7b3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:41.092309 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet a071981a53d741ce84778a3a6f7b3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:41.092640 9864 tablet_replica.cc:330] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:41.092700 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet a071981a53d741ce84778a3a6f7b3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:41.092819 9864 raft_consensus.cc:2229] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.092847 10212 tablet_replica.cc:330] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:41.092985 9864 raft_consensus.cc:2258] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.093168 10212 raft_consensus.cc:2229] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.093344 10212 raft_consensus.cc:2258] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.093631 9864 ts_tablet_manager.cc:1875] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.094275 10212 ts_tablet_manager.cc:1875] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.095320 9864 ts_tablet_manager.cc:1888] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.095399 9864 log.cc:1196] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a071981a53d741ce84778a3a6f7b3a01
I20240501 07:58:41.095669 9864 ts_tablet_manager.cc:1909] T a071981a53d741ce84778a3a6f7b3a01 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:41.095991 10213 tablet_replica.cc:330] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:41.096166 10213 raft_consensus.cc:2229] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.096501 10213 raft_consensus.cc:2258] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.096501 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table_new_name [id=0370673905e24aedaaa54798f6e7a849]) successfully deleted
I20240501 07:58:41.096686 10212 ts_tablet_manager.cc:1888] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.096894 10212 log.cc:1196] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a071981a53d741ce84778a3a6f7b3a01
I20240501 07:58:41.097347 10212 ts_tablet_manager.cc:1909] T a071981a53d741ce84778a3a6f7b3a01 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:41.097623 10213 ts_tablet_manager.cc:1875] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.098599 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table_new_name [id=0370673905e24aedaaa54798f6e7a849]) successfully deleted
I20240501 07:58:41.099547 10213 ts_tablet_manager.cc:1888] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.099618 10213 log.cc:1196] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a071981a53d741ce84778a3a6f7b3a01
I20240501 07:58:41.099938 10213 ts_tablet_manager.cc:1909] T a071981a53d741ce84778a3a6f7b3a01 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:41.100589 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet a071981a53d741ce84778a3a6f7b3a01 (table rebalancer_dynamic_table_new_name [id=0370673905e24aedaaa54798f6e7a849]) successfully deleted
I20240501 07:58:41.113159 7009 consensus_queue.cc:237] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.114429 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:41.114593 7271 raft_consensus.cc:1261] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.115288 9932 consensus_queue.cc:1035] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:41.115746 6946 consensus_peers.cc:485] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48964a6c57584c3eacc6dffac29d77a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.116134 6878 raft_consensus.cc:1261] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.116508 9932 consensus_queue.cc:1035] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.119074 10020 raft_consensus.cc:2932] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.119446 6878 raft_consensus.cc:2932] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.122152 7271 raft_consensus.cc:2932] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.122717 7252 tablet_service.cc:1467] Processing CreateTablet for tablet bccd5e4f0ef448bd856edf6fa2521631 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.123008 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccd5e4f0ef448bd856edf6fa2521631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.123080 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2d5511dc0d4e8cb6d71ca894da4884 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.123338 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2d5511dc0d4e8cb6d71ca894da4884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.123374 7121 tablet_service.cc:1467] Processing CreateTablet for tablet bccd5e4f0ef448bd856edf6fa2521631 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.123631 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccd5e4f0ef448bd856edf6fa2521631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.125062 10232 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:41.125613 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2d5511dc0d4e8cb6d71ca894da4884 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.125897 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2d5511dc0d4e8cb6d71ca894da4884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.126075 10232 tablet_bootstrap.cc:654] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.126076 6858 tablet_service.cc:1467] Processing CreateTablet for tablet bccd5e4f0ef448bd856edf6fa2521631 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.126350 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bccd5e4f0ef448bd856edf6fa2521631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.126765 10233 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.127228 10232 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:41.127312 10232 ts_tablet_manager.cc:1367] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:41.127797 10233 tablet_bootstrap.cc:654] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.127725 10232 raft_consensus.cc:348] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.127843 10232 raft_consensus.cc:374] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.127887 10232 raft_consensus.cc:729] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.128048 10232 consensus_queue.cc:260] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.128422 10232 ts_tablet_manager.cc:1398] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:41.128604 9646 tablet_service.cc:1467] Processing CreateTablet for tablet ee29c58cdc6544e8933300ad5faecb3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.128702 10238 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:41.128850 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee29c58cdc6544e8933300ad5faecb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.128885 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2d5511dc0d4e8cb6d71ca894da4884 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.129074 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2d5511dc0d4e8cb6d71ca894da4884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.129494 10238 tablet_bootstrap.cc:654] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.130498 10238 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:41.130584 10238 ts_tablet_manager.cc:1367] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.130968 10238 raft_consensus.cc:348] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.131475 9507 tablet_service.cc:1467] Processing CreateTablet for tablet ee29c58cdc6544e8933300ad5faecb3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.131726 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee29c58cdc6544e8933300ad5faecb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.131963 10236 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.132236 6729 catalog_manager.cc:5484] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.132324 7252 tablet_service.cc:1467] Processing CreateTablet for tablet ee29c58cdc6544e8933300ad5faecb3a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.132573 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee29c58cdc6544e8933300ad5faecb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.132920 10236 tablet_bootstrap.cc:654] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.134450 10233 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:41.134549 10233 ts_tablet_manager.cc:1367] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:41.135210 10233 raft_consensus.cc:348] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.135366 10233 raft_consensus.cc:374] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.135448 10233 raft_consensus.cc:729] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.131086 10238 raft_consensus.cc:374] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.135808 10238 raft_consensus.cc:729] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.135941 10238 consensus_queue.cc:260] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.136222 10238 ts_tablet_manager.cc:1398] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:58:41.136384 10238 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:41.137177 10238 tablet_bootstrap.cc:654] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.135653 10233 consensus_queue.cc:260] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.138741 10236 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:41.139377 10236 ts_tablet_manager.cc:1367] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:41.139389 10246 ts_tablet_manager.cc:902] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.139889 10236 raft_consensus.cc:348] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.140085 10236 raft_consensus.cc:374] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.140087 10246 tablet_copy_client.cc:308] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.140281 10236 raft_consensus.cc:729] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.140708 10236 consensus_queue.cc:260] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.140772 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet de03fdd63bd347dea4eb6e5d410da68e from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:52369)
I20240501 07:58:41.140897 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet de03fdd63bd347dea4eb6e5d410da68e from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:52369: session id = fa6964452854415a85ad72f8639768d6-de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.142237 7030 tablet_copy_source_session.cc:205] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.144348 10238 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:41.144428 10238 ts_tablet_manager.cc:1367] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 07:58:41.144672 10236 ts_tablet_manager.cc:1398] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:41.144729 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de03fdd63bd347dea4eb6e5d410da68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.144870 10238 raft_consensus.cc:348] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.144958 10238 raft_consensus.cc:374] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.144989 10238 raft_consensus.cc:729] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.144941 9879 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.145092 10238 consensus_queue.cc:260] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.145370 10238 ts_tablet_manager.cc:1398] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:41.148459 10246 tablet_copy_client.cc:783] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.148687 10246 tablet_copy_client.cc:647] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.149312 9879 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.151124 10236 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.153066 10232 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:41.153456 10236 tablet_bootstrap.cc:654] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.154071 10232 tablet_bootstrap.cc:654] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.154170 10233 ts_tablet_manager.cc:1398] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20240501 07:58:41.154474 10236 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:41.154541 10236 ts_tablet_manager.cc:1367] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:41.154732 10233 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.155318 10236 raft_consensus.cc:348] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.155424 10236 raft_consensus.cc:374] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.155463 10236 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.155593 10236 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.155951 10236 ts_tablet_manager.cc:1398] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.158643 10233 tablet_bootstrap.cc:654] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.159749 10233 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:41.159739 10234 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:41.159797 10232 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:41.159883 10232 ts_tablet_manager.cc:1367] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:41.160342 10232 raft_consensus.cc:348] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.160462 10232 raft_consensus.cc:374] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.160506 10232 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.160588 10233 ts_tablet_manager.cc:1367] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:41.160638 10232 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.160773 10234 tablet_bootstrap.cc:654] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.160997 10232 ts_tablet_manager.cc:1398] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.161051 10246 tablet_copy_client.cc:523] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.161083 10233 raft_consensus.cc:348] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.161204 10233 raft_consensus.cc:374] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.161264 10233 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.161422 10233 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.161717 10233 ts_tablet_manager.cc:1398] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.164731 10246 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.171629 10234 tablet_bootstrap.cc:492] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:41.171733 10234 ts_tablet_manager.cc:1367] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:41.172142 10234 raft_consensus.cc:348] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.172243 10234 raft_consensus.cc:374] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.172273 10234 raft_consensus.cc:729] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.172400 10234 consensus_queue.cc:260] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.172679 10234 ts_tablet_manager.cc:1398] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:41.184841 6878 consensus_queue.cc:237] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.186357 7009 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.186357 7139 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.186832 9863 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.187227 9863 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:41.187600 6814 consensus_peers.cc:485] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2304f5e5d09a47b793cbb4f60b255525. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.188879 10001 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.189406 7139 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.191749 7009 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.192344 6730 catalog_manager.cc:5484] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.195598 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:41.202344 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]), partition=
I20240501 07:58:41.202512 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]), partition=
I20240501 07:58:41.202623 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.202741 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.203714 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]), partition=
I20240501 07:58:41.203949 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.205628 10280 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.206137 10234 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:41.206640 10280 tablet_bootstrap.cc:654] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.207232 10234 tablet_bootstrap.cc:654] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.208276 10280 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:41.208365 10280 ts_tablet_manager.cc:1367] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:41.208734 10246 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.208866 10280 raft_consensus.cc:348] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.208990 10246 tablet_bootstrap.cc:492] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:41.208989 10280 raft_consensus.cc:374] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.209075 10280 raft_consensus.cc:729] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.209219 10280 consensus_queue.cc:260] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.209529 10246 ts_tablet_manager.cc:1367] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.005s
I20240501 07:58:41.209662 10280 ts_tablet_manager.cc:1398] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.209925 10246 raft_consensus.cc:348] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.210023 10246 raft_consensus.cc:729] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.210134 10246 consensus_queue.cc:260] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.210398 10246 ts_tablet_manager.cc:1398] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:41.210569 10234 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:41.210644 10234 ts_tablet_manager.cc:1367] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:41.211061 10234 raft_consensus.cc:348] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.211149 10234 raft_consensus.cc:374] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.211176 10234 raft_consensus.cc:729] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.211277 10234 consensus_queue.cc:260] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.211427 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-de03fdd63bd347dea4eb6e5d410da68e received from {username='slave'} at 127.0.99.134:52369
I20240501 07:58:41.211520 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session fa6964452854415a85ad72f8639768d6-de03fdd63bd347dea4eb6e5d410da68e on tablet de03fdd63bd347dea4eb6e5d410da68e with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:41.211530 10234 ts_tablet_manager.cc:1398] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.214258 10279 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.219429 10279 tablet_bootstrap.cc:654] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.220620 10279 tablet_bootstrap.cc:492] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:41.220715 10279 ts_tablet_manager.cc:1367] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:41.221187 10279 raft_consensus.cc:348] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.221357 10279 raft_consensus.cc:374] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.221415 10279 raft_consensus.cc:729] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.221596 10279 consensus_queue.cc:260] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.221891 10279 ts_tablet_manager.cc:1398] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.226529 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 07:58:41.226603 10294 ts_tablet_manager.cc:902] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.226656 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 07:58:41.228032 10294 tablet_copy_client.cc:308] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.230608 7271 consensus_queue.cc:237] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.231130 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet 1f5dd2edfd7b4e28b226299dc2a4fc04
I20240501 07:58:41.231518 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet d5df0c0e783e491e8b37dd349df4ad34
I20240501 07:58:41.231770 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 74c1f8c60d0344c381aebc78f198944d
I20240501 07:58:41.232651 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.232700 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:41.232833 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.232913 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c1f8c60d0344c381aebc78f198944d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.233034 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:41.233187 9868 tablet_replica.cc:330] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.233198 9864 tablet_replica.cc:330] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:41.233330 9864 raft_consensus.cc:2229] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.233330 9868 raft_consensus.cc:2229] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.233449 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:41.233476 9864 raft_consensus.cc:2258] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.233520 9868 raft_consensus.cc:2258] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.233678 6878 raft_consensus.cc:1261] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:41.234091 7207 consensus_peers.cc:485] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4b3a231d67e4092a94d3539e31e62f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.234287 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:41.234522 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c1f8c60d0344c381aebc78f198944d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:41.234515 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c1f8c60d0344c381aebc78f198944d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:41.234645 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet d5df0c0e783e491e8b37dd349df4ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:41.234694 10041 raft_consensus.cc:482] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:41.234755 10041 raft_consensus.cc:504] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.235352 7009 raft_consensus.cc:1261] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.235570 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 33e711b7a9684137969ce2bbffae40c2 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53657)
I20240501 07:58:41.235661 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 33e711b7a9684137969ce2bbffae40c2 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53657: session id = ac6c8bc1b74b483791e262abee7862cf-33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:41.235728 10041 leader_election.cc:288] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:41.236109 9868 ts_tablet_manager.cc:1875] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.236107 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccd5e4f0ef448bd856edf6fa2521631" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:41.236225 7139 raft_consensus.cc:2454] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:41.236519 7030 tablet_copy_source_session.cc:205] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.236506 7206 leader_election.cc:302] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:41.236955 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccd5e4f0ef448bd856edf6fa2521631" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:41.237069 6878 raft_consensus.cc:2454] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:41.237350 10042 raft_consensus.cc:2781] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:41.237401 10042 raft_consensus.cc:482] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:41.237442 10042 raft_consensus.cc:3037] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.237851 10294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e711b7a9684137969ce2bbffae40c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.238294 10042 raft_consensus.cc:504] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.238339 9868 ts_tablet_manager.cc:1888] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.238399 9868 log.cc:1196] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f5dd2edfd7b4e28b226299dc2a4fc04
I20240501 07:58:41.238669 10042 leader_election.cc:288] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:41.238687 9868 ts_tablet_manager.cc:1909] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.238754 10042 consensus_queue.cc:1035] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.235585 9864 ts_tablet_manager.cc:1875] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.239356 9868 tablet_replica.cc:330] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.239480 9868 raft_consensus.cc:2229] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.239588 9868 raft_consensus.cc:2258] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.239650 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccd5e4f0ef448bd856edf6fa2521631" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:41.239745 7139 raft_consensus.cc:3037] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.239988 9868 ts_tablet_manager.cc:1875] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.240697 7139 raft_consensus.cc:2454] T bccd5e4f0ef448bd856edf6fa2521631 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:41.241060 7206 leader_election.cc:302] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:41.241312 9864 ts_tablet_manager.cc:1888] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.241374 9864 log.cc:1196] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1f5dd2edfd7b4e28b226299dc2a4fc04
I20240501 07:58:41.241412 9796 raft_consensus.cc:2781] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:41.241480 9796 raft_consensus.cc:686] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:41.241652 9864 ts_tablet_manager.cc:1909] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:41.241641 9868 ts_tablet_manager.cc:1888] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.241623 9796 consensus_queue.cc:237] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.241698 9868 log.cc:1196] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d5df0c0e783e491e8b37dd349df4ad34
I20240501 07:58:41.241941 9868 ts_tablet_manager.cc:1909] T d5df0c0e783e491e8b37dd349df4ad34 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.242167 9864 tablet_replica.cc:330] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:41.242364 9868 tablet_replica.cc:330] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.242471 9868 raft_consensus.cc:2229] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.242558 9868 raft_consensus.cc:2258] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.242923 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bccd5e4f0ef448bd856edf6fa2521631" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:41.243034 6877 raft_consensus.cc:3037] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.243660 10041 consensus_queue.cc:1035] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.243676 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.243790 6877 raft_consensus.cc:2454] T bccd5e4f0ef448bd856edf6fa2521631 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:41.243847 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.243953 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet d5df0c0e783e491e8b37dd349df4ad34 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.244024 9868 ts_tablet_manager.cc:1875] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.244740 9864 raft_consensus.cc:2229] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.245251 9864 raft_consensus.cc:2258] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.246466 6737 catalog_manager.cc:5484] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:41.246604 10303 tablet_replica.cc:330] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:41.246726 10303 raft_consensus.cc:2229] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.246872 10303 raft_consensus.cc:2258] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.247342 10303 ts_tablet_manager.cc:1875] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.248231 9868 ts_tablet_manager.cc:1888] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.248342 9868 log.cc:1196] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/74c1f8c60d0344c381aebc78f198944d
I20240501 07:58:41.248641 9868 ts_tablet_manager.cc:1909] T 74c1f8c60d0344c381aebc78f198944d P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.249333 10294 tablet_copy_client.cc:783] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.249575 10294 tablet_copy_client.cc:647] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.249892 10303 ts_tablet_manager.cc:1888] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.249984 10303 log.cc:1196] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/74c1f8c60d0344c381aebc78f198944d
I20240501 07:58:41.250640 10307 ts_tablet_manager.cc:902] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.250716 10303 ts_tablet_manager.cc:1909] T 74c1f8c60d0344c381aebc78f198944d P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:41.251096 10294 tablet_copy_client.cc:523] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.251421 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 74c1f8c60d0344c381aebc78f198944d (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.251441 10307 tablet_copy_client.cc:308] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.251626 10303 tablet_replica.cc:330] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:41.251813 10303 raft_consensus.cc:2229] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.252197 10303 raft_consensus.cc:2258] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.252735 10303 ts_tablet_manager.cc:1875] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.253293 10294 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.253458 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 74c1f8c60d0344c381aebc78f198944d (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.254498 9864 ts_tablet_manager.cc:1875] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.255620 10303 ts_tablet_manager.cc:1888] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.255743 10303 log.cc:1196] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5df0c0e783e491e8b37dd349df4ad34
I20240501 07:58:41.256043 10303 ts_tablet_manager.cc:1909] T d5df0c0e783e491e8b37dd349df4ad34 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:41.256667 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet d5df0c0e783e491e8b37dd349df4ad34 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.256763 9864 ts_tablet_manager.cc:1888] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.256829 9864 log.cc:1196] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d5df0c0e783e491e8b37dd349df4ad34
I20240501 07:58:41.256929 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 58aab13671d647e485f9be94283b5239 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:39427)
I20240501 07:58:41.257038 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 58aab13671d647e485f9be94283b5239 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:39427: session id = 7fa3a88c25964445975bb47e0feed460-58aab13671d647e485f9be94283b5239
I20240501 07:58:41.257179 9864 ts_tablet_manager.cc:1909] T d5df0c0e783e491e8b37dd349df4ad34 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:41.257810 7030 tablet_copy_source_session.cc:205] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.257803 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet d5df0c0e783e491e8b37dd349df4ad34 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.258538 10302 tablet_replica.cc:330] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:41.259054 10302 raft_consensus.cc:2229] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.259258 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58aab13671d647e485f9be94283b5239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.259673 10042 raft_consensus.cc:2932] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.260063 6876 raft_consensus.cc:2932] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.261930 7009 raft_consensus.cc:2932] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.263949 10307 tablet_copy_client.cc:783] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.264124 10307 tablet_copy_client.cc:647] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.264247 6729 catalog_manager.cc:5484] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.268047 10307 tablet_copy_client.cc:523] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.269608 10302 raft_consensus.cc:2258] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 07:58:41.270335 6723 catalog_manager.cc:4797] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): delete failed for tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 because the tablet was not found. No further retry: Not found: Tablet not found: 1f5dd2edfd7b4e28b226299dc2a4fc04
I20240501 07:58:41.270401 10315 ts_tablet_manager.cc:902] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:41.270628 10302 ts_tablet_manager.cc:1875] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.271153 10307 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.271149 10315 tablet_copy_client.cc:308] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:41.273659 10302 ts_tablet_manager.cc:1888] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.273730 10302 log.cc:1196] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1f5dd2edfd7b4e28b226299dc2a4fc04
I20240501 07:58:41.274598 10320 ts_tablet_manager.cc:902] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.275681 10320 tablet_copy_client.cc:308] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.276751 10302 ts_tablet_manager.cc:1909] T 1f5dd2edfd7b4e28b226299dc2a4fc04 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:41.277529 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 1f5dd2edfd7b4e28b226299dc2a4fc04 (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.277565 10302 tablet_replica.cc:330] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:41.277774 10302 raft_consensus.cc:2229] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.277839 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 1671c966902c4eb98b238c4419c525fe from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:49987)
I20240501 07:58:41.277964 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 1671c966902c4eb98b238c4419c525fe from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:49987: session id = 49adb9e99822494196ed22044bfed81f-1671c966902c4eb98b238c4419c525fe
I20240501 07:58:41.279245 10302 raft_consensus.cc:2258] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.279423 7292 tablet_copy_source_session.cc:205] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.279886 10302 ts_tablet_manager.cc:1875] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.281066 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1671c966902c4eb98b238c4419c525fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.283468 10315 tablet_copy_client.cc:783] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.283486 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet e05498632f0945c29f723a4613a26a04 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53675)
I20240501 07:58:41.283604 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet e05498632f0945c29f723a4613a26a04 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53675: session id = ac6c8bc1b74b483791e262abee7862cf-e05498632f0945c29f723a4613a26a04
I20240501 07:58:41.283747 10302 ts_tablet_manager.cc:1888] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.283799 10302 log.cc:1196] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/74c1f8c60d0344c381aebc78f198944d
I20240501 07:58:41.284051 10302 ts_tablet_manager.cc:1909] T 74c1f8c60d0344c381aebc78f198944d P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:41.284871 7161 tablet_copy_source_session.cc:205] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.284976 10315 tablet_copy_client.cc:647] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.286583 10315 tablet_copy_client.cc:523] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.287194 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05498632f0945c29f723a4613a26a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.287530 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 74c1f8c60d0344c381aebc78f198944d (table rebalance_test_table_4 [id=9d3a5debb62c4bb2a70cb35425c5d2ef]) successfully deleted
I20240501 07:58:41.288944 10315 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.289176 10320 tablet_copy_client.cc:783] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.289317 10320 tablet_copy_client.cc:647] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.294965 10294 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.295162 10320 tablet_copy_client.cc:523] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.295344 10294 tablet_bootstrap.cc:492] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:41.295578 10294 ts_tablet_manager.cc:1367] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.042s user 0.022s sys 0.008s
I20240501 07:58:41.296075 10294 raft_consensus.cc:348] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.296386 10320 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.296175 10294 raft_consensus.cc:729] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.296803 10294 consensus_queue.cc:260] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.297217 10294 ts_tablet_manager.cc:1398] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.298588 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-33e711b7a9684137969ce2bbffae40c2 received from {username='slave'} at 127.0.99.135:53657
I20240501 07:58:41.298660 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-33e711b7a9684137969ce2bbffae40c2 on tablet 33e711b7a9684137969ce2bbffae40c2 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:41.312095 7139 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.313520 7270 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:41.313714 7076 consensus_peers.cc:485] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80152bb09e4e42a09e106cf8b58178e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.313927 7009 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.314054 9972 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.314553 10307 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.314777 9972 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.314910 10307 tablet_bootstrap.cc:492] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:41.315203 10307 ts_tablet_manager.cc:1367] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.044s user 0.034s sys 0.000s
I20240501 07:58:41.315842 10307 raft_consensus.cc:348] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.316022 10307 raft_consensus.cc:729] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.316242 10307 consensus_queue.cc:260] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.316704 10307 ts_tablet_manager.cc:1398] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.317968 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-58aab13671d647e485f9be94283b5239 received from {username='slave'} at 127.0.99.136:39427
I20240501 07:58:41.318055 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-58aab13671d647e485f9be94283b5239 on tablet 58aab13671d647e485f9be94283b5239 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:41.319525 10346 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.321192 7271 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.323836 7009 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.325449 6730 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:41.334725 10320 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.335163 10320 tablet_bootstrap.cc:492] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:41.335381 10320 ts_tablet_manager.cc:1367] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.001s
I20240501 07:58:41.335866 10320 raft_consensus.cc:348] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.336125 10320 raft_consensus.cc:729] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.336313 10320 consensus_queue.cc:260] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.336917 10320 ts_tablet_manager.cc:1398] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:41.337473 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-e05498632f0945c29f723a4613a26a04 received from {username='slave'} at 127.0.99.135:53675
I20240501 07:58:41.337590 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-e05498632f0945c29f723a4613a26a04 on tablet e05498632f0945c29f723a4613a26a04 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:41.348285 7139 consensus_queue.cc:237] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.349284 9734 raft_consensus.cc:482] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:41.349390 9734 raft_consensus.cc:504] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.349702 10315 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.349880 9734 leader_election.cc:288] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.350378 7271 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.350477 10315 tablet_bootstrap.cc:492] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
W20240501 07:58:41.350656 7077 consensus_peers.cc:485] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f334b66a84b94c3ba15004253b4cb714. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.350996 6877 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.350975 9942 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.351279 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e50fb82504b8c99fb8d4f1a07fbf5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:41.351425 9266 raft_consensus.cc:2454] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:41.351723 10315 ts_tablet_manager.cc:1367] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.002s
I20240501 07:58:41.352167 10315 raft_consensus.cc:348] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.352289 10315 raft_consensus.cc:729] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.352411 10315 consensus_queue.cc:260] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.352584 9338 leader_election.cc:302] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:41.352699 10315 ts_tablet_manager.cc:1398] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.353356 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-1671c966902c4eb98b238c4419c525fe received from {username='slave'} at 127.0.99.133:49987
I20240501 07:58:41.353449 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 49adb9e99822494196ed22044bfed81f-1671c966902c4eb98b238c4419c525fe on tablet 1671c966902c4eb98b238c4419c525fe with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:41.353804 9942 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.355145 9734 raft_consensus.cc:2781] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:41.355208 9734 raft_consensus.cc:482] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:41.355245 9734 raft_consensus.cc:3037] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.356078 9734 raft_consensus.cc:504] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.356441 9734 leader_election.cc:288] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.357210 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e50fb82504b8c99fb8d4f1a07fbf5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:41.357331 9266 raft_consensus.cc:3037] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.358322 9266 raft_consensus.cc:2454] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:41.358305 9972 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.358661 9338 leader_election.cc:302] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:41.359097 9734 raft_consensus.cc:2781] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:41.359200 9734 raft_consensus.cc:686] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:41.359380 9734 consensus_queue.cc:237] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:41.361748 7271 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.361800 6730 catalog_manager.cc:5484] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:41.362646 6876 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.364673 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e50fb82504b8c99fb8d4f1a07fbf5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:41.364815 7139 raft_consensus.cc:2454] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:41.364925 6737 catalog_manager.cc:5484] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.365010 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e50fb82504b8c99fb8d4f1a07fbf5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:41.365099 7141 raft_consensus.cc:3037] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.366065 7141 raft_consensus.cc:2454] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:41.382787 9735 raft_consensus.cc:482] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:41.382895 9735 raft_consensus.cc:504] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.383271 9735 leader_election.cc:288] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:41.383786 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:41.383808 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:41.383917 6876 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:41.383972 9665 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:41.384388 9470 leader_election.cc:302] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:41.384889 9735 raft_consensus.cc:2781] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:41.384945 9735 raft_consensus.cc:482] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:41.384982 9735 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.385942 9735 raft_consensus.cc:504] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.386231 9735 leader_election.cc:288] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:41.386997 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:41.387148 6876 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.387168 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:41.387265 9665 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.388135 9665 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:41.388142 6876 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:41.388870 9470 leader_election.cc:302] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:41.389077 9735 raft_consensus.cc:2781] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:41.389227 9735 raft_consensus.cc:686] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:41.389353 9735 consensus_queue.cc:237] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.391239 7009 consensus_queue.cc:237] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.391248 6729 catalog_manager.cc:5484] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:41.392650 7139 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.393018 7271 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.393075 9936 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.393844 9936 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:41.394235 6945 consensus_peers.cc:485] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 651beb63e03d40bca88f01ebed416754. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.395623 9932 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.396060 7271 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.398895 6729 catalog_manager.cc:5484] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.399583 7139 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:41.406736 10314 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:41.406854 10314 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.407371 10314 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:41.407697 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:41.407858 9665 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:41.408432 7207 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:41.408584 10314 raft_consensus.cc:2781] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:41.408623 10314 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:41.408655 10314 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.409626 10314 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.409950 10314 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:41.410467 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:41.410595 9665 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.411497 9665 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:41.412089 7207 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:41.412254 10314 raft_consensus.cc:2781] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:41.412322 10314 raft_consensus.cc:686] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:41.412439 10314 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:41.415728 6737 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:41.416586 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:41.416586 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:41.416824 9535 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:41.423312 7271 consensus_queue.cc:237] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.424589 7139 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.424628 6876 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:41.424799 7207 consensus_peers.cc:485] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e7f613e16c245de83fde8c277b8d350. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.425004 9796 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.425304 10314 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.427160 9796 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.427580 7139 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.428870 6876 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.430197 6737 catalog_manager.cc:5484] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:41.441922 10320 ts_tablet_manager.cc:902] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.442411 10320 tablet_copy_client.cc:308] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.442871 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet c80723cb0ce4446483a5f1754e066081 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53675)
I20240501 07:58:41.442970 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet c80723cb0ce4446483a5f1754e066081 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53675: session id = ac6c8bc1b74b483791e262abee7862cf-c80723cb0ce4446483a5f1754e066081
I20240501 07:58:41.443908 7161 tablet_copy_source_session.cc:205] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.445010 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80723cb0ce4446483a5f1754e066081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.447190 10320 tablet_copy_client.cc:783] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.447371 10320 tablet_copy_client.cc:647] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.448479 10320 tablet_copy_client.cc:523] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.449580 10320 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.472618 10320 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.472901 10320 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:41.473071 10320 ts_tablet_manager.cc:1367] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20240501 07:58:41.473493 10320 raft_consensus.cc:348] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.473619 10320 raft_consensus.cc:729] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.473763 10320 consensus_queue.cc:260] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.474133 10320 ts_tablet_manager.cc:1398] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:58:41.474779 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-c80723cb0ce4446483a5f1754e066081 received from {username='slave'} at 127.0.99.135:53675
I20240501 07:58:41.474869 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-c80723cb0ce4446483a5f1754e066081 on tablet c80723cb0ce4446483a5f1754e066081 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:41.478729 10246 ts_tablet_manager.cc:902] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.479257 10246 tablet_copy_client.cc:308] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.480585 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 0d3d410c640243848a478aa2605c1bf0 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:52369)
I20240501 07:58:41.480690 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 0d3d410c640243848a478aa2605c1bf0 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:52369: session id = fa6964452854415a85ad72f8639768d6-0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:41.481683 7030 tablet_copy_source_session.cc:205] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.482468 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3d410c640243848a478aa2605c1bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.482461 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:41.487138 7139 raft_consensus.cc:1261] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:41.487257 9266 raft_consensus.cc:1261] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:41.487665 9882 consensus_queue.cc:1035] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.487951 9734 consensus_queue.cc:1035] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.488023 10246 tablet_copy_client.cc:783] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.488195 10246 tablet_copy_client.cc:647] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.490321 10246 tablet_copy_client.cc:523] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.491353 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]). Expected version 1 got 0
I20240501 07:58:41.491782 10246 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.492604 6734 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]). Expected version 1 got 0
I20240501 07:58:41.492810 10462 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:41.494001 10462 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:41.494913 6734 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]). Expected version 1 got 0
I20240501 07:58:41.495066 10463 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:41.495862 10463 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:41.501318 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c] alter complete (version 1)
I20240501 07:58:41.502708 10315 ts_tablet_manager.cc:902] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:41.503275 10315 tablet_copy_client.cc:308] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:41.505193 10469 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:41.505535 10469 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:41.509310 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c]). Expected version 1 got 0
I20240501 07:58:41.509332 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 4882718ff90c4d939917f4970a9acb9e from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:48159)
I20240501 07:58:41.509433 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 4882718ff90c4d939917f4970a9acb9e from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:48159: session id = 49adb9e99822494196ed22044bfed81f-4882718ff90c4d939917f4970a9acb9e
I20240501 07:58:41.510763 6899 tablet_copy_source_session.cc:205] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.512637 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4882718ff90c4d939917f4970a9acb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.514861 10315 tablet_copy_client.cc:783] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.516625 10315 tablet_copy_client.cc:647] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.518244 10315 tablet_copy_client.cc:523] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.520946 10315 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.525126 10320 ts_tablet_manager.cc:902] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.525521 10320 tablet_copy_client.cc:308] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.526448 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 48964a6c57584c3eacc6dffac29d77a8 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53657)
I20240501 07:58:41.526561 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 48964a6c57584c3eacc6dffac29d77a8 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53657: session id = ac6c8bc1b74b483791e262abee7862cf-48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:41.527809 7030 tablet_copy_source_session.cc:205] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.528375 9403 raft_consensus.cc:1203] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.528659 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48964a6c57584c3eacc6dffac29d77a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.533053 10320 tablet_copy_client.cc:783] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.533892 10320 tablet_copy_client.cc:647] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.537560 10320 tablet_copy_client.cc:523] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.540019 10320 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.548005 10246 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.548384 10246 tablet_bootstrap.cc:492] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:41.548624 10246 ts_tablet_manager.cc:1367] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.057s user 0.031s sys 0.001s
I20240501 07:58:41.549163 10246 raft_consensus.cc:348] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.549371 10246 raft_consensus.cc:729] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.549595 10246 consensus_queue.cc:260] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.551167 10475 ts_tablet_manager.cc:902] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.551800 10475 tablet_copy_client.cc:308] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.552196 10246 ts_tablet_manager.cc:1398] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:41.558017 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 37e73deb03274bba82ceb67087208bdd from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:58:41.558127 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 37e73deb03274bba82ceb67087208bdd from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-37e73deb03274bba82ceb67087208bdd
I20240501 07:58:41.559092 7161 tablet_copy_source_session.cc:205] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.561213 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e73deb03274bba82ceb67087208bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.565224 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-0d3d410c640243848a478aa2605c1bf0 received from {username='slave'} at 127.0.99.134:52369
I20240501 07:58:41.565313 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session fa6964452854415a85ad72f8639768d6-0d3d410c640243848a478aa2605c1bf0 on tablet 0d3d410c640243848a478aa2605c1bf0 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:41.566967 10307 ts_tablet_manager.cc:902] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:41.567425 10307 tablet_copy_client.cc:308] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:41.574344 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet ab7dcdd38eff43fe9fff9081ee644c8d from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:42067)
I20240501 07:58:41.574468 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet ab7dcdd38eff43fe9fff9081ee644c8d from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:42067: session id = 7fa3a88c25964445975bb47e0feed460-ab7dcdd38eff43fe9fff9081ee644c8d
I20240501 07:58:41.575548 7292 tablet_copy_source_session.cc:205] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.575548 9507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:41.575757 9382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:41.577260 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7dcdd38eff43fe9fff9081ee644c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.579666 10307 tablet_copy_client.cc:783] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.579849 10307 tablet_copy_client.cc:647] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.580054 9646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:41.584528 6989 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:41.595575 7120 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:41.595687 10307 tablet_copy_client.cc:523] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.596691 10246 ts_tablet_manager.cc:902] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:41.597260 10246 tablet_copy_client.cc:308] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:41.599264 10307 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.603531 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:41.604317 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 43b82e035ecb4e23973a2163e15c1dac from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:47957)
I20240501 07:58:41.604406 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 43b82e035ecb4e23973a2163e15c1dac from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:47957: session id = fa6964452854415a85ad72f8639768d6-43b82e035ecb4e23973a2163e15c1dac
I20240501 07:58:41.605332 6899 tablet_copy_source_session.cc:205] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.610977 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b82e035ecb4e23973a2163e15c1dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.613325 10462 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:41.613636 10462 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:41.614688 10463 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:41.614691 10469 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:41.614966 10469 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:41.614969 10463 tablet.cc:1620] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:41.616690 10246 tablet_copy_client.cc:783] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.616873 10246 tablet_copy_client.cc:647] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.616966 10475 tablet_copy_client.cc:783] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.617102 10475 tablet_copy_client.cc:647] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.619283 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=935d253a0f2d412bbbd41380ed57c38c] alter complete (version 2)
I20240501 07:58:41.620743 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:41.621151 10475 tablet_copy_client.cc:523] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.624752 10246 tablet_copy_client.cc:523] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.626045 10246 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.624577 10475 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.630983 7252 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:41.635959 9238 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:41.642771 10320 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.644991 10320 tablet_bootstrap.cc:492] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:41.649703 10504 ts_tablet_manager.cc:902] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:41.650041 10320 ts_tablet_manager.cc:1367] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.110s user 0.026s sys 0.005s
I20240501 07:58:41.650316 10504 tablet_copy_client.cc:308] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:41.650489 10320 raft_consensus.cc:348] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.650619 10320 raft_consensus.cc:729] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.650749 10320 consensus_queue.cc:260] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.651053 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet f4b3a231d67e4092a94d3539e31e62f4 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:42067)
I20240501 07:58:41.651103 10320 ts_tablet_manager.cc:1398] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.651162 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet f4b3a231d67e4092a94d3539e31e62f4 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:42067: session id = 7fa3a88c25964445975bb47e0feed460-f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:41.652132 7292 tablet_copy_source_session.cc:205] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.654136 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-48964a6c57584c3eacc6dffac29d77a8 received from {username='slave'} at 127.0.99.135:53657
I20240501 07:58:41.654217 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-48964a6c57584c3eacc6dffac29d77a8 on tablet 48964a6c57584c3eacc6dffac29d77a8 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:41.654479 10314 consensus_queue.cc:1035] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.664155 10315 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.664445 10315 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:41.664635 10315 ts_tablet_manager.cc:1367] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.144s user 0.024s sys 0.007s
I20240501 07:58:41.665380 10042 consensus_queue.cc:1035] T bccd5e4f0ef448bd856edf6fa2521631 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.673375 10504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b3a231d67e4092a94d3539e31e62f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.676926 10498 ts_tablet_manager.cc:902] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:41.677424 10498 tablet_copy_client.cc:308] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:41.678056 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 2304f5e5d09a47b793cbb4f60b255525 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:47957)
I20240501 07:58:41.678156 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 2304f5e5d09a47b793cbb4f60b255525 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:47957: session id = fa6964452854415a85ad72f8639768d6-2304f5e5d09a47b793cbb4f60b255525
I20240501 07:58:41.678546 10504 tablet_copy_client.cc:783] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.678707 10504 tablet_copy_client.cc:647] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.679256 6899 tablet_copy_source_session.cc:205] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.679785 10504 tablet_copy_client.cc:523] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.680899 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2304f5e5d09a47b793cbb4f60b255525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.682169 10504 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.684022 10307 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.684058 10498 tablet_copy_client.cc:783] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.684193 10498 tablet_copy_client.cc:647] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.684269 10307 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:41.684412 10307 ts_tablet_manager.cc:1367] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.085s user 0.027s sys 0.000s
I20240501 07:58:41.685432 10307 raft_consensus.cc:348] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.685665 10307 raft_consensus.cc:729] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.685844 10307 consensus_queue.cc:260] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.686506 10307 ts_tablet_manager.cc:1398] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.686805 10498 tablet_copy_client.cc:523] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.686925 10519 ts_tablet_manager.cc:902] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.688019 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-ab7dcdd38eff43fe9fff9081ee644c8d received from {username='slave'} at 127.0.99.136:42067
I20240501 07:58:41.688066 10498 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.688114 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-ab7dcdd38eff43fe9fff9081ee644c8d on tablet ab7dcdd38eff43fe9fff9081ee644c8d with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:41.688248 10519 tablet_copy_client.cc:308] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.688635 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 00cac44dfd0f4d97a3056b590748ba58 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:58:41.688719 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 00cac44dfd0f4d97a3056b590748ba58 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-00cac44dfd0f4d97a3056b590748ba58
I20240501 07:58:41.689553 7161 tablet_copy_source_session.cc:205] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.690383 10519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cac44dfd0f4d97a3056b590748ba58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.692863 10519 tablet_copy_client.cc:783] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.693018 10519 tablet_copy_client.cc:647] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.693126 10315 raft_consensus.cc:348] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.698230 9534 raft_consensus.cc:1203] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.694051 10519 tablet_copy_client.cc:523] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.703227 10315 raft_consensus.cc:729] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.703461 10315 consensus_queue.cc:260] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.703892 10315 ts_tablet_manager.cc:1398] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240501 07:58:41.704514 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-4882718ff90c4d939917f4970a9acb9e received from {username='slave'} at 127.0.99.133:48159
I20240501 07:58:41.704566 10519 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.704610 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session 49adb9e99822494196ed22044bfed81f-4882718ff90c4d939917f4970a9acb9e on tablet 4882718ff90c4d939917f4970a9acb9e with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:41.707768 10315 ts_tablet_manager.cc:902] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.708250 10315 tablet_copy_client.cc:308] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.708654 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 80152bb09e4e42a09e106cf8b58178e3 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:58:41.708756 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 80152bb09e4e42a09e106cf8b58178e3 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-80152bb09e4e42a09e106cf8b58178e3
I20240501 07:58:41.709673 7161 tablet_copy_source_session.cc:205] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.710491 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80152bb09e4e42a09e106cf8b58178e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.721022 10504 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.721326 10504 tablet_bootstrap.cc:492] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:41.721540 10504 ts_tablet_manager.cc:1367] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.039s user 0.024s sys 0.000s
I20240501 07:58:41.722070 10504 raft_consensus.cc:348] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.722203 10504 raft_consensus.cc:729] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.722364 10504 consensus_queue.cc:260] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.722420 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:41.723098 10504 ts_tablet_manager.cc:1398] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:58:41.723623 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-f4b3a231d67e4092a94d3539e31e62f4 received from {username='slave'} at 127.0.99.136:42067
I20240501 07:58:41.723711 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-f4b3a231d67e4092a94d3539e31e62f4 on tablet f4b3a231d67e4092a94d3539e31e62f4 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:41.728821 10462 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:41.729871 10475 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.730230 10475 tablet_bootstrap.cc:492] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:41.730507 10475 ts_tablet_manager.cc:1367] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.106s user 0.021s sys 0.007s
I20240501 07:58:41.730684 10246 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.730789 10463 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:41.730942 10246 tablet_bootstrap.cc:492] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:41.731141 10246 ts_tablet_manager.cc:1367] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.105s user 0.024s sys 0.004s
I20240501 07:58:41.731323 10475 raft_consensus.cc:348] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.731491 10475 raft_consensus.cc:729] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.731578 10246 raft_consensus.cc:348] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.731683 10246 raft_consensus.cc:729] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.731729 10475 consensus_queue.cc:260] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.732236 10475 ts_tablet_manager.cc:1398] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.733657 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-37e73deb03274bba82ceb67087208bdd received from {username='slave'} at 127.0.99.133:44859
I20240501 07:58:41.733740 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-37e73deb03274bba82ceb67087208bdd on tablet 37e73deb03274bba82ceb67087208bdd with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:41.734032 10469 tablet.cc:1722] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:41.734562 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=935d253a0f2d412bbbd41380ed57c38c] alter complete (version 3)
I20240501 07:58:41.734680 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 07:58:41.734774 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 07:58:41.736816 10246 consensus_queue.cc:260] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.737321 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 0305b131804a4bc096dfe6cd801f98ee
I20240501 07:58:41.737591 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet fecf2f8d747d4b2694a63034b86635b6
I20240501 07:58:41.737758 10246 ts_tablet_manager.cc:1398] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:41.737795 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.738422 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 0305b131804a4bc096dfe6cd801f98ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:41.738504 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-43b82e035ecb4e23973a2163e15c1dac received from {username='slave'} at 127.0.99.134:47957
I20240501 07:58:41.738576 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session fa6964452854415a85ad72f8639768d6-43b82e035ecb4e23973a2163e15c1dac on tablet 43b82e035ecb4e23973a2163e15c1dac with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:41.738646 9864 tablet_replica.cc:330] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:41.738780 9864 raft_consensus.cc:2229] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.738811 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet de03fdd63bd347dea4eb6e5d410da68e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:41.738950 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 0305b131804a4bc096dfe6cd801f98ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.738971 9864 raft_consensus.cc:2258] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.738960 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet de03fdd63bd347dea4eb6e5d410da68e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:41.739097 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf2f8d747d4b2694a63034b86635b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.739233 9868 tablet_replica.cc:330] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.739365 9868 raft_consensus.cc:2229] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.739491 9868 raft_consensus.cc:2258] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.739491 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0305b131804a4bc096dfe6cd801f98ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:41.739631 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf2f8d747d4b2694a63034b86635b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:41.739720 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf2f8d747d4b2694a63034b86635b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:41.739789 10303 tablet_replica.cc:330] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:41.739892 10302 tablet_replica.cc:330] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:41.739920 10303 raft_consensus.cc:2229] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.739993 10302 raft_consensus.cc:2229] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.740046 10303 raft_consensus.cc:2258] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.740064 9868 ts_tablet_manager.cc:1875] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.740106 10302 raft_consensus.cc:2258] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.740190 10543 tablet_replica.cc:330] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:41.740288 10543 raft_consensus.cc:2229] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.740372 10543 raft_consensus.cc:2258] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.740517 10302 ts_tablet_manager.cc:1875] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.740514 10303 ts_tablet_manager.cc:1875] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.740955 10543 ts_tablet_manager.cc:1875] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.742126 9868 ts_tablet_manager.cc:1888] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.742188 9868 log.cc:1196] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0305b131804a4bc096dfe6cd801f98ee
I20240501 07:58:41.742449 9868 ts_tablet_manager.cc:1909] T 0305b131804a4bc096dfe6cd801f98ee P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.742457 10303 ts_tablet_manager.cc:1888] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.742457 10302 ts_tablet_manager.cc:1888] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.742527 10302 log.cc:1196] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fecf2f8d747d4b2694a63034b86635b6
I20240501 07:58:41.742527 10303 log.cc:1196] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0305b131804a4bc096dfe6cd801f98ee
I20240501 07:58:41.742780 10302 ts_tablet_manager.cc:1909] T fecf2f8d747d4b2694a63034b86635b6 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:41.742784 10303 ts_tablet_manager.cc:1909] T 0305b131804a4bc096dfe6cd801f98ee P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:41.739634 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet de03fdd63bd347dea4eb6e5d410da68e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:41.743083 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet de03fdd63bd347dea4eb6e5d410da68e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:41.743165 10543 ts_tablet_manager.cc:1888] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.743223 10543 log.cc:1196] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.743239 9868 tablet_replica.cc:330] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.743371 9868 raft_consensus.cc:2229] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.743419 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 0305b131804a4bc096dfe6cd801f98ee (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.743480 10543 ts_tablet_manager.cc:1909] T de03fdd63bd347dea4eb6e5d410da68e P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:41.743492 9868 raft_consensus.cc:2258] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.743836 9864 ts_tablet_manager.cc:1875] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.744030 9868 ts_tablet_manager.cc:1875] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.744529 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet de03fdd63bd347dea4eb6e5d410da68e (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.745601 9868 ts_tablet_manager.cc:1888] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.745651 9868 log.cc:1196] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fecf2f8d747d4b2694a63034b86635b6
I20240501 07:58:41.745831 9868 ts_tablet_manager.cc:1909] T fecf2f8d747d4b2694a63034b86635b6 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.746019 9864 ts_tablet_manager.cc:1888] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.746088 9864 log.cc:1196] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0305b131804a4bc096dfe6cd801f98ee
I20240501 07:58:41.746326 10303 tablet_replica.cc:330] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:41.746344 9864 ts_tablet_manager.cc:1909] T 0305b131804a4bc096dfe6cd801f98ee P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:41.746477 10303 raft_consensus.cc:2229] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.746575 9868 tablet_replica.cc:330] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:41.746644 10303 raft_consensus.cc:2258] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.746762 9868 raft_consensus.cc:2229] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.746908 9864 tablet_replica.cc:330] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:41.747047 9868 raft_consensus.cc:2258] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.747054 9864 raft_consensus.cc:2229] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.747047 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 0305b131804a4bc096dfe6cd801f98ee (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.747189 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet fecf2f8d747d4b2694a63034b86635b6 (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.747198 10315 tablet_copy_client.cc:783] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.747442 10315 tablet_copy_client.cc:647] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.747462 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 0305b131804a4bc096dfe6cd801f98ee (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.747573 10303 ts_tablet_manager.cc:1875] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.748284 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet fecf2f8d747d4b2694a63034b86635b6 (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.749943 10303 ts_tablet_manager.cc:1888] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:41.750012 10303 log.cc:1196] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fecf2f8d747d4b2694a63034b86635b6
I20240501 07:58:41.750289 10303 ts_tablet_manager.cc:1909] T fecf2f8d747d4b2694a63034b86635b6 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:41.751036 10315 tablet_copy_client.cc:523] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.751078 10303 tablet_replica.cc:330] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:41.751283 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet fecf2f8d747d4b2694a63034b86635b6 (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.751298 10303 raft_consensus.cc:2229] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.751534 10303 raft_consensus.cc:2258] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.752195 10315 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.753472 10303 ts_tablet_manager.cc:1875] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.755180 10303 ts_tablet_manager.cc:1888] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.755293 10303 log.cc:1196] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.755576 10303 ts_tablet_manager.cc:1909] T de03fdd63bd347dea4eb6e5d410da68e P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:41.756258 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet de03fdd63bd347dea4eb6e5d410da68e (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.758821 9864 raft_consensus.cc:2258] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.759531 9864 ts_tablet_manager.cc:1875] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.761524 9864 ts_tablet_manager.cc:1888] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.761644 9864 log.cc:1196] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.762006 9864 ts_tablet_manager.cc:1909] T de03fdd63bd347dea4eb6e5d410da68e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:41.763167 9868 ts_tablet_manager.cc:1875] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.765162 9868 ts_tablet_manager.cc:1888] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.765316 9868 log.cc:1196] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/de03fdd63bd347dea4eb6e5d410da68e
I20240501 07:58:41.765627 9868 ts_tablet_manager.cc:1909] T de03fdd63bd347dea4eb6e5d410da68e P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:41.766299 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet de03fdd63bd347dea4eb6e5d410da68e (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.766337 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet de03fdd63bd347dea4eb6e5d410da68e (table rebalance_test_table_6 [id=dae68e56a81f491399cd850fbef7cd2e]) successfully deleted
I20240501 07:58:41.771490 10498 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.771754 10498 tablet_bootstrap.cc:492] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:41.771920 10498 ts_tablet_manager.cc:1367] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.001s
I20240501 07:58:41.772351 10498 raft_consensus.cc:348] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.772478 10498 raft_consensus.cc:729] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.772630 10498 consensus_queue.cc:260] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.777675 10498 ts_tablet_manager.cc:1398] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:41.779320 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-2304f5e5d09a47b793cbb4f60b255525 received from {username='slave'} at 127.0.99.134:47957
I20240501 07:58:41.779421 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session fa6964452854415a85ad72f8639768d6-2304f5e5d09a47b793cbb4f60b255525 on tablet 2304f5e5d09a47b793cbb4f60b255525 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:41.780138 10384 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.782169 10384 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.784335 10519 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.799219 10519 tablet_bootstrap.cc:492] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:41.799436 10519 ts_tablet_manager.cc:1367] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.095s user 0.025s sys 0.000s
I20240501 07:58:41.799974 10519 raft_consensus.cc:348] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.800101 10519 raft_consensus.cc:729] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.800232 10519 consensus_queue.cc:260] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.800544 10519 ts_tablet_manager.cc:1398] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.802711 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-00cac44dfd0f4d97a3056b590748ba58 received from {username='slave'} at 127.0.99.133:44859
I20240501 07:58:41.802827 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-00cac44dfd0f4d97a3056b590748ba58 on tablet 00cac44dfd0f4d97a3056b590748ba58 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:41.804006 10315 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.804281 10315 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:41.804433 10315 ts_tablet_manager.cc:1367] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.052s user 0.025s sys 0.000s
I20240501 07:58:41.805023 10315 raft_consensus.cc:348] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.805130 10315 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.805253 10315 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.805509 10315 ts_tablet_manager.cc:1398] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.806006 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-80152bb09e4e42a09e106cf8b58178e3 received from {username='slave'} at 127.0.99.133:44859
I20240501 07:58:41.806087 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-80152bb09e4e42a09e106cf8b58178e3 on tablet 80152bb09e4e42a09e106cf8b58178e3 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:41.818938 10041 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.826714 9535 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.828545 9534 raft_consensus.cc:1203] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.830439 10041 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:41.836522 9665 raft_consensus.cc:1203] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.837805 10498 ts_tablet_manager.cc:902] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:41.838253 10498 tablet_copy_client.cc:308] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:41.838670 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 651beb63e03d40bca88f01ebed416754 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:52369)
I20240501 07:58:41.838655 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:41.838762 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 651beb63e03d40bca88f01ebed416754 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:52369: session id = fa6964452854415a85ad72f8639768d6-651beb63e03d40bca88f01ebed416754
I20240501 07:58:41.838781 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:41.839744 7030 tablet_copy_source_session.cc:205] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.840657 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651beb63e03d40bca88f01ebed416754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.841004 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5
I20240501 07:58:41.841795 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:41.841977 10302 tablet_replica.cc:330] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:41.842121 10302 raft_consensus.cc:2229] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.842224 10302 raft_consensus.cc:2258] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.842744 10498 tablet_copy_client.cc:783] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.842775 10302 ts_tablet_manager.cc:1875] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.842882 10498 tablet_copy_client.cc:647] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.843323 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:41.843369 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:41 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:41.843518 10543 tablet_replica.cc:330] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:41.843613 10543 raft_consensus.cc:2229] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:41.843767 10543 raft_consensus.cc:2258] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.844798 10586 tablet_replica.cc:330] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:41.844938 10302 ts_tablet_manager.cc:1888] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.844972 10586 raft_consensus.cc:2229] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.845007 10302 log.cc:1196] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a8e50fb82504b8c99fb8d4f1a07fbf5
I20240501 07:58:41.845083 10543 ts_tablet_manager.cc:1875] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.845095 10586 raft_consensus.cc:2258] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.845269 10302 ts_tablet_manager.cc:1909] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:41.846164 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table_new_name [id=935d253a0f2d412bbbd41380ed57c38c]) successfully deleted
I20240501 07:58:41.846808 10543 ts_tablet_manager.cc:1888] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.846872 10543 log.cc:1196] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5a8e50fb82504b8c99fb8d4f1a07fbf5
I20240501 07:58:41.847164 10543 ts_tablet_manager.cc:1909] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:41.847529 10586 ts_tablet_manager.cc:1875] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:41.847765 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table_new_name [id=935d253a0f2d412bbbd41380ed57c38c]) successfully deleted
I20240501 07:58:41.848253 9534 raft_consensus.cc:1203] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.848613 10498 tablet_copy_client.cc:523] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.849853 10498 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.850741 10586 ts_tablet_manager.cc:1888] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:41.850817 10586 log.cc:1196] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5a8e50fb82504b8c99fb8d4f1a07fbf5
I20240501 07:58:41.851152 10586 ts_tablet_manager.cc:1909] T 5a8e50fb82504b8c99fb8d4f1a07fbf5 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:41.852084 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 5a8e50fb82504b8c99fb8d4f1a07fbf5 (table rebalancer_dynamic_table_new_name [id=935d253a0f2d412bbbd41380ed57c38c]) successfully deleted
I20240501 07:58:41.857409 9266 raft_consensus.cc:1203] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.864472 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:41.876132 9507 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:41.878580 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:41.883188 6990 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:41.883239 10498 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.883853 10498 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:41.884042 10498 ts_tablet_manager.cc:1367] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20240501 07:58:41.884497 10498 raft_consensus.cc:348] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.884696 10498 raft_consensus.cc:729] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.884878 10498 consensus_queue.cc:260] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.885689 10498 ts_tablet_manager.cc:1398] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.891304 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-651beb63e03d40bca88f01ebed416754 received from {username='slave'} at 127.0.99.134:52369
I20240501 07:58:41.891398 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session fa6964452854415a85ad72f8639768d6-651beb63e03d40bca88f01ebed416754 on tablet 651beb63e03d40bca88f01ebed416754 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:41.892302 9382 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:41.892429 7120 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:41.895522 9238 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:41.895565 9646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:41.909245 10320 ts_tablet_manager.cc:902] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:41.909768 10320 tablet_copy_client.cc:308] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:41.914309 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet f334b66a84b94c3ba15004253b4cb714 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53675)
I20240501 07:58:41.914433 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet f334b66a84b94c3ba15004253b4cb714 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53675: session id = ac6c8bc1b74b483791e262abee7862cf-f334b66a84b94c3ba15004253b4cb714
I20240501 07:58:41.915473 7161 tablet_copy_source_session.cc:205] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.916477 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f334b66a84b94c3ba15004253b4cb714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.918816 10320 tablet_copy_client.cc:783] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.918985 10320 tablet_copy_client.cc:647] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.920459 10320 tablet_copy_client.cc:523] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.921772 10320 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.942715 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:41.945109 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:41.949308 9507 tablet_service.cc:1467] Processing CreateTablet for tablet ae59405a605c4b30a508002a9df30945 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.949405 9382 tablet_service.cc:1467] Processing CreateTablet for tablet ae59405a605c4b30a508002a9df30945 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.949605 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59405a605c4b30a508002a9df30945. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.949692 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59405a605c4b30a508002a9df30945. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.951495 9238 tablet_service.cc:1467] Processing CreateTablet for tablet ae59405a605c4b30a508002a9df30945 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.951758 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59405a605c4b30a508002a9df30945. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.951858 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3e4cd27e4545baa255a01023cf1d68 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.952041 10320 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.952082 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3e4cd27e4545baa255a01023cf1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.952272 10320 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:41.952430 10320 ts_tablet_manager.cc:1367] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.000s
I20240501 07:58:41.952826 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3e4cd27e4545baa255a01023cf1d68 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.952876 10320 raft_consensus.cc:348] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.952977 10320 raft_consensus.cc:729] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.953083 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3e4cd27e4545baa255a01023cf1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.953132 10320 consensus_queue.cc:260] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:41.954180 10320 ts_tablet_manager.cc:1398] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:41.954313 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 010c12e2cc194cdb87ff9299a8e42a77 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.954591 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010c12e2cc194cdb87ff9299a8e42a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.954613 7161 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-f334b66a84b94c3ba15004253b4cb714 received from {username='slave'} at 127.0.99.135:53675
I20240501 07:58:41.954679 7161 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-f334b66a84b94c3ba15004253b4cb714 on tablet f334b66a84b94c3ba15004253b4cb714 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:41.955233 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3e4cd27e4545baa255a01023cf1d68 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.955233 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 010c12e2cc194cdb87ff9299a8e42a77 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.955438 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3e4cd27e4545baa255a01023cf1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.955438 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010c12e2cc194cdb87ff9299a8e42a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.956485 10611 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:41.957369 10611 tablet_bootstrap.cc:654] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.957840 10609 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.958448 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 010c12e2cc194cdb87ff9299a8e42a77 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:41.958711 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010c12e2cc194cdb87ff9299a8e42a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.958890 10609 tablet_bootstrap.cc:654] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.959476 10611 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:41.959559 10611 ts_tablet_manager.cc:1367] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:41.960150 10611 raft_consensus.cc:348] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.960258 10611 raft_consensus.cc:374] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.960291 10611 raft_consensus.cc:729] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.960400 10611 consensus_queue.cc:260] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.960724 10611 ts_tablet_manager.cc:1398] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:41.961045 10610 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.961931 10610 tablet_bootstrap.cc:654] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.962404 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]), partition=
I20240501 07:58:41.962589 10613 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.962622 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377c905ed6343bfa102b3aa9699e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.963582 10613 tablet_bootstrap.cc:654] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.964867 10613 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:41.964897 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]), partition=
I20240501 07:58:41.964947 10613 ts_tablet_manager.cc:1367] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:41.965099 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377c905ed6343bfa102b3aa9699e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.965332 10613 raft_consensus.cc:348] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.965431 10613 raft_consensus.cc:374] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.965462 10613 raft_consensus.cc:729] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.965620 10613 consensus_queue.cc:260] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.965862 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]), partition=
I20240501 07:58:41.965956 10613 ts_tablet_manager.cc:1398] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.966050 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377c905ed6343bfa102b3aa9699e8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.966109 10613 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.966850 10613 tablet_bootstrap.cc:654] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.967505 10616 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.968474 10616 tablet_bootstrap.cc:654] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.969208 10613 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:41.969341 10613 ts_tablet_manager.cc:1367] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:41.970204 10613 raft_consensus.cc:348] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.970358 10613 raft_consensus.cc:374] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.970431 10613 raft_consensus.cc:729] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.970573 10613 consensus_queue.cc:260] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.970695 9665 raft_consensus.cc:1203] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.970882 10609 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:41.970971 10609 ts_tablet_manager.cc:1367] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:41.971325 10613 ts_tablet_manager.cc:1398] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:41.971473 10613 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:41.971417 10609 raft_consensus.cc:348] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.971547 10609 raft_consensus.cc:374] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.971635 10609 raft_consensus.cc:729] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.971783 10609 consensus_queue.cc:260] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.972337 10613 tablet_bootstrap.cc:654] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.973165 10609 ts_tablet_manager.cc:1398] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.973345 10609 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.974397 10609 tablet_bootstrap.cc:654] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.974766 10616 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:41.974861 10616 ts_tablet_manager.cc:1367] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:41.975410 10616 raft_consensus.cc:348] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.975526 10616 raft_consensus.cc:374] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.975577 10616 raft_consensus.cc:729] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.975764 10616 consensus_queue.cc:260] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.976716 10609 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:41.976819 10609 ts_tablet_manager.cc:1367] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:41.977056 10616 ts_tablet_manager.cc:1398] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.977209 10616 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:41.977218 10609 raft_consensus.cc:348] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.977406 10609 raft_consensus.cc:374] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.977497 10609 raft_consensus.cc:729] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.977699 10609 consensus_queue.cc:260] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.978075 10609 ts_tablet_manager.cc:1398] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.978374 10616 tablet_bootstrap.cc:654] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.978534 10609 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:41.979763 10609 tablet_bootstrap.cc:654] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.981165 10609 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:41.981294 10609 ts_tablet_manager.cc:1367] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:41.981621 10610 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:41.981745 10610 ts_tablet_manager.cc:1367] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240501 07:58:41.981889 10609 raft_consensus.cc:348] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.982064 10609 raft_consensus.cc:374] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.982148 10609 raft_consensus.cc:729] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.982304 10609 consensus_queue.cc:260] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.982434 10610 raft_consensus.cc:348] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.982522 10610 raft_consensus.cc:374] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.982575 10610 raft_consensus.cc:729] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.982697 10610 consensus_queue.cc:260] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:41.982776 10609 ts_tablet_manager.cc:1398] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.983060 10610 ts_tablet_manager.cc:1398] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:41.983291 10610 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.983294 9877 raft_consensus.cc:482] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:41.983381 9877 raft_consensus.cc:504] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.983842 9877 leader_election.cc:288] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:41.984272 10610 tablet_bootstrap.cc:654] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.985394 10610 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:41.985456 10610 ts_tablet_manager.cc:1367] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:41.985517 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3e4cd27e4545baa255a01023cf1d68" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:41.985785 10504 ts_tablet_manager.cc:902] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:41.985862 10610 raft_consensus.cc:348] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.985939 10610 raft_consensus.cc:374] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.985970 10610 raft_consensus.cc:729] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.986120 10610 consensus_queue.cc:260] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.986268 10504 tablet_copy_client.cc:308] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:41.986402 10610 ts_tablet_manager.cc:1398] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.986558 10610 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:41.986675 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 4e7f613e16c245de83fde8c277b8d350 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:42067)
I20240501 07:58:41.986779 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 4e7f613e16c245de83fde8c277b8d350 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:42067: session id = 7fa3a88c25964445975bb47e0feed460-4e7f613e16c245de83fde8c277b8d350
I20240501 07:58:41.987406 10610 tablet_bootstrap.cc:654] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:41.987764 7292 tablet_copy_source_session.cc:205] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.988060 9266 raft_consensus.cc:2454] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:41.988579 10610 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:41.988576 10613 tablet_bootstrap.cc:492] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:41.988631 10610 ts_tablet_manager.cc:1367] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:41.988662 10613 ts_tablet_manager.cc:1367] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20240501 07:58:41.988946 10610 raft_consensus.cc:348] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.989010 10610 raft_consensus.cc:374] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.989032 10610 raft_consensus.cc:729] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.989157 10610 consensus_queue.cc:260] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.989169 10613 raft_consensus.cc:348] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.989274 10613 raft_consensus.cc:374] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.989306 10613 raft_consensus.cc:729] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.989413 10613 consensus_queue.cc:260] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:41.989491 10610 ts_tablet_manager.cc:1398] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.989521 10504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f613e16c245de83fde8c277b8d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.989682 10613 ts_tablet_manager.cc:1398] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:41.990164 10616 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:41.990236 10616 ts_tablet_manager.cc:1367] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:41.990491 9601 leader_election.cc:302] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:41.990702 10616 raft_consensus.cc:348] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.990819 10616 raft_consensus.cc:374] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:41.990854 10616 raft_consensus.cc:729] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:41.990958 10616 consensus_queue.cc:260] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:41.991348 10616 ts_tablet_manager.cc:1398] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:41.991654 10504 tablet_copy_client.cc:783] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.991797 10504 tablet_copy_client.cc:647] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.992718 9266 raft_consensus.cc:1203] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.993918 9403 raft_consensus.cc:1203] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.994258 9877 raft_consensus.cc:2781] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:41.994310 9877 raft_consensus.cc:482] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:41.994354 9877 raft_consensus.cc:3037] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.994391 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3e4cd27e4545baa255a01023cf1d68" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:41.994522 9534 raft_consensus.cc:2454] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:41.995240 9877 raft_consensus.cc:504] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.995579 9877 leader_election.cc:288] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:41.996321 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3e4cd27e4545baa255a01023cf1d68" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:41.996450 9266 raft_consensus.cc:3037] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.997543 9266 raft_consensus.cc:2454] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:41.997745 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3e4cd27e4545baa255a01023cf1d68" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:41.997829 9601 leader_election.cc:302] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:41.997845 9534 raft_consensus.cc:3037] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:41.997975 9877 raft_consensus.cc:2781] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:41.998064 9877 raft_consensus.cc:686] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:41.998380 9877 consensus_queue.cc:237] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:41.998929 9534 raft_consensus.cc:2454] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:41.999650 10504 tablet_copy_client.cc:523] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:42.001009 10504 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:42.004226 6734 catalog_manager.cc:5484] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:42.014168 9403 raft_consensus.cc:1203] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.031154 10253 raft_consensus.cc:482] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.031261 10253 raft_consensus.cc:504] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:42.031677 10253 leader_election.cc:288] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.032181 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377c905ed6343bfa102b3aa9699e8e9" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:42.032326 9403 raft_consensus.cc:2454] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:42.033840 9206 leader_election.cc:302] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.033948 9266 raft_consensus.cc:1203] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.035229 10253 raft_consensus.cc:2781] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.035285 10253 raft_consensus.cc:482] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.035318 10253 raft_consensus.cc:3037] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.036334 10253 raft_consensus.cc:504] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:42.036667 10253 leader_election.cc:288] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.037546 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377c905ed6343bfa102b3aa9699e8e9" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:42.037665 9403 raft_consensus.cc:3037] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.038667 9403 raft_consensus.cc:2454] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:42.039127 9206 leader_election.cc:302] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.039285 10253 raft_consensus.cc:2781] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.039348 10253 raft_consensus.cc:686] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:42.039479 10253 consensus_queue.cc:237] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:42.041610 6734 catalog_manager.cc:5484] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:42.042253 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377c905ed6343bfa102b3aa9699e8e9" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:42.042379 9534 raft_consensus.cc:2454] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:42.042735 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377c905ed6343bfa102b3aa9699e8e9" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:42.042848 9535 raft_consensus.cc:3037] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.044194 9535 raft_consensus.cc:2454] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:42.044580 10504 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:42.044943 10504 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:42.045222 10504 ts_tablet_manager.cc:1367] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.044s user 0.023s sys 0.004s
I20240501 07:58:42.045784 10504 raft_consensus.cc:348] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.045975 10504 raft_consensus.cc:729] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.046203 10504 consensus_queue.cc:260] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.046625 10504 ts_tablet_manager.cc:1398] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:42.047287 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-4e7f613e16c245de83fde8c277b8d350 received from {username='slave'} at 127.0.99.136:42067
I20240501 07:58:42.047369 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-4e7f613e16c245de83fde8c277b8d350 on tablet 4e7f613e16c245de83fde8c277b8d350 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:42.047931 10648 raft_consensus.cc:482] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.048084 10648 raft_consensus.cc:504] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.048960 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59405a605c4b30a508002a9df30945" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:42.049103 9403 raft_consensus.cc:2454] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:42.049520 9206 leader_election.cc:302] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.049697 10253 raft_consensus.cc:2781] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.049741 10253 raft_consensus.cc:482] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.049784 10253 raft_consensus.cc:3037] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.050771 9882 raft_consensus.cc:482] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.050837 10253 raft_consensus.cc:504] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.050860 9882 raft_consensus.cc:504] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:42.051126 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59405a605c4b30a508002a9df30945" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:42.051257 9535 raft_consensus.cc:2454] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:42.051355 9882 leader_election.cc:288] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:42.052001 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010c12e2cc194cdb87ff9299a8e42a77" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:42.052099 10253 leader_election.cc:288] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.052116 9665 raft_consensus.cc:2454] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:42.052328 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010c12e2cc194cdb87ff9299a8e42a77" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:42.052455 6876 raft_consensus.cc:2454] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:42.052564 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59405a605c4b30a508002a9df30945" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:42.052675 9535 raft_consensus.cc:3037] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.052744 9338 leader_election.cc:302] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.053005 9882 raft_consensus.cc:2781] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.053062 9882 raft_consensus.cc:482] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.053099 9882 raft_consensus.cc:3037] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.053508 9535 raft_consensus.cc:2454] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:42.053907 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59405a605c4b30a508002a9df30945" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:42.053998 9403 raft_consensus.cc:3037] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.054057 9882 raft_consensus.cc:504] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:42.054376 9882 leader_election.cc:288] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:42.054682 9207 leader_election.cc:302] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:42.054803 9403 raft_consensus.cc:2454] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:42.055258 10648 leader_election.cc:288] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.055308 10253 raft_consensus.cc:2781] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.055399 10253 raft_consensus.cc:686] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:42.055542 10253 consensus_queue.cc:237] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.055992 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010c12e2cc194cdb87ff9299a8e42a77" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:42.056102 6876 raft_consensus.cc:3037] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.057030 6876 raft_consensus.cc:2454] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:42.057328 9338 leader_election.cc:302] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.057512 9882 raft_consensus.cc:2781] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.057587 9882 raft_consensus.cc:686] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:42.057716 9882 consensus_queue.cc:237] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:42.058254 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010c12e2cc194cdb87ff9299a8e42a77" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:42.058357 9665 raft_consensus.cc:3037] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.059329 9665 raft_consensus.cc:2454] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:42.060024 6729 catalog_manager.cc:5484] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.060074 6732 catalog_manager.cc:5484] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:42.069449 9403 raft_consensus.cc:1203] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.075417 9535 raft_consensus.cc:1203] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.099730 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:42.102970 9238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:42.103260 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:42.114097 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:42.120406 7120 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:42.122946 9515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.127727 9383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.149428 6990 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:42.159178 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:42.163867 9403 raft_consensus.cc:1261] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:42.163967 9535 raft_consensus.cc:1261] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:42.164400 10253 consensus_queue.cc:1035] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.164592 10648 consensus_queue.cc:1035] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.167551 6733 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]). Expected version 1 got 0
I20240501 07:58:42.168593 6730 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]). Expected version 1 got 0
I20240501 07:58:42.169005 10469 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:42.169323 10469 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:42.169739 9266 raft_consensus.cc:1203] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.170370 10462 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:42.170646 10462 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:42.174607 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747] alter complete (version 1)
I20240501 07:58:42.188895 10692 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:42.189397 10692 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:42.190280 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747]). Expected version 1 got 0
I20240501 07:58:42.209760 7009 consensus_queue.cc:237] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.211417 9403 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.211623 6876 raft_consensus.cc:1261] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.212086 9936 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.212347 9936 consensus_queue.cc:1035] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.214042 10020 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.215636 6876 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.217098 9403 raft_consensus.cc:2932] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.217928 6729 catalog_manager.cc:5484] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.221381 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3d410c640243848a478aa2605c1bf0 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.221596 10303 tablet_replica.cc:330] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.221729 10303 raft_consensus.cc:2229] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.221971 10303 raft_consensus.cc:2258] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.222560 10303 ts_tablet_manager.cc:1875] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:42.224431 10303 ts_tablet_manager.cc:1888] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:42.224507 10303 log.cc:1196] T 0d3d410c640243848a478aa2605c1bf0 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:42.225040 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 0d3d410c640243848a478aa2605c1bf0 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:42.239701 9665 raft_consensus.cc:1203] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.244311 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 07:58:42.244451 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 07:58:42.246753 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 08a748c1a5574f5faf7966f733ff2844
I20240501 07:58:42.247081 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 6f251432f70a417fa8be862182b16d91
I20240501 07:58:42.247361 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 338a7d3de39641278a7268d61ba78795
I20240501 07:58:42.247876 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 338a7d3de39641278a7268d61ba78795 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.247876 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a748c1a5574f5faf7966f733ff2844 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:42.248225 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f251432f70a417fa8be862182b16d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.248327 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a748c1a5574f5faf7966f733ff2844 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:42.248402 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 338a7d3de39641278a7268d61ba78795 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:42.248490 9864 tablet_replica.cc:330] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:42.248564 10302 tablet_replica.cc:330] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:42.248600 10303 tablet_replica.cc:330] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.248631 9864 raft_consensus.cc:2229] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.248709 10303 raft_consensus.cc:2229] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.248705 10302 raft_consensus.cc:2229] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.248781 9864 raft_consensus.cc:2258] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.248807 10303 raft_consensus.cc:2258] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.248850 10302 raft_consensus.cc:2258] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.249279 10303 ts_tablet_manager.cc:1875] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.249280 9864 ts_tablet_manager.cc:1875] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.249401 10302 ts_tablet_manager.cc:1875] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.251191 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f251432f70a417fa8be862182b16d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:42.251464 10302 ts_tablet_manager.cc:1888] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.251541 10302 log.cc:1196] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/08a748c1a5574f5faf7966f733ff2844
I20240501 07:58:42.251550 9864 ts_tablet_manager.cc:1888] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.251613 9864 log.cc:1196] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/08a748c1a5574f5faf7966f733ff2844
I20240501 07:58:42.251598 10303 ts_tablet_manager.cc:1888] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.251655 10303 log.cc:1196] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/338a7d3de39641278a7268d61ba78795
I20240501 07:58:42.251896 10302 ts_tablet_manager.cc:1909] T 08a748c1a5574f5faf7966f733ff2844 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:42.251905 9864 ts_tablet_manager.cc:1909] T 08a748c1a5574f5faf7966f733ff2844 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:42.251971 10303 ts_tablet_manager.cc:1909] T 338a7d3de39641278a7268d61ba78795 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:42.252456 10303 tablet_replica.cc:330] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.252610 10303 raft_consensus.cc:2229] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.252619 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 338a7d3de39641278a7268d61ba78795 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.252760 10303 raft_consensus.cc:2258] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.253216 10303 ts_tablet_manager.cc:1875] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.253222 9403 raft_consensus.cc:1203] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.253341 10302 tablet_replica.cc:330] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:42.253520 10302 raft_consensus.cc:2229] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.253875 10302 raft_consensus.cc:2258] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.254410 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 08a748c1a5574f5faf7966f733ff2844 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.255093 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a748c1a5574f5faf7966f733ff2844 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.255266 9864 tablet_replica.cc:330] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:42.255255 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f251432f70a417fa8be862182b16d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.255373 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 338a7d3de39641278a7268d61ba78795 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.255477 9868 tablet_replica.cc:330] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.255584 9868 raft_consensus.cc:2229] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.255712 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 08a748c1a5574f5faf7966f733ff2844 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.255740 9868 raft_consensus.cc:2258] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.256203 9868 ts_tablet_manager.cc:1875] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.257100 9864 raft_consensus.cc:2229] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.256925 10302 ts_tablet_manager.cc:1875] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.257301 9864 raft_consensus.cc:2258] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.257747 9864 ts_tablet_manager.cc:1875] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.258244 9868 ts_tablet_manager.cc:1888] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.258301 9868 log.cc:1196] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/08a748c1a5574f5faf7966f733ff2844
I20240501 07:58:42.258606 9868 ts_tablet_manager.cc:1909] T 08a748c1a5574f5faf7966f733ff2844 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.259182 9868 tablet_replica.cc:330] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.259258 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 08a748c1a5574f5faf7966f733ff2844 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.259310 9868 raft_consensus.cc:2229] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.259420 9868 raft_consensus.cc:2258] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.259446 10302 ts_tablet_manager.cc:1888] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.259865 9868 ts_tablet_manager.cc:1875] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.259873 10302 log.cc:1196] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/338a7d3de39641278a7268d61ba78795
I20240501 07:58:42.260125 9864 ts_tablet_manager.cc:1888] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.260183 9864 log.cc:1196] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f251432f70a417fa8be862182b16d91
I20240501 07:58:42.260440 9864 ts_tablet_manager.cc:1909] T 6f251432f70a417fa8be862182b16d91 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:42.260532 10302 ts_tablet_manager.cc:1909] T 338a7d3de39641278a7268d61ba78795 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:42.261087 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 6f251432f70a417fa8be862182b16d91 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.261117 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 338a7d3de39641278a7268d61ba78795 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.261665 9868 ts_tablet_manager.cc:1888] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.261710 9868 log.cc:1196] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f251432f70a417fa8be862182b16d91
I20240501 07:58:42.261945 9868 ts_tablet_manager.cc:1909] T 6f251432f70a417fa8be862182b16d91 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.262269 10303 ts_tablet_manager.cc:1888] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.262363 10303 log.cc:1196] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f251432f70a417fa8be862182b16d91
I20240501 07:58:42.262562 10303 ts_tablet_manager.cc:1909] T 6f251432f70a417fa8be862182b16d91 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:42.263257 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 6f251432f70a417fa8be862182b16d91 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.263525 9868 tablet_replica.cc:330] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.263562 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 6f251432f70a417fa8be862182b16d91 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.263653 9868 raft_consensus.cc:2229] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.263963 9868 raft_consensus.cc:2258] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.264418 9868 ts_tablet_manager.cc:1875] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.265883 9868 ts_tablet_manager.cc:1888] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.265941 9868 log.cc:1196] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/338a7d3de39641278a7268d61ba78795
I20240501 07:58:42.266166 9868 ts_tablet_manager.cc:1909] T 338a7d3de39641278a7268d61ba78795 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.266422 9266 raft_consensus.cc:1203] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.266816 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 338a7d3de39641278a7268d61ba78795 (table rebalance_test_table_8 [id=5bf16c03ff40418ab1a63483d52907a8]) successfully deleted
I20240501 07:58:42.277128 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:42.282723 10469 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:42.282747 10462 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:42.283048 10462 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:42.285346 10692 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:42.285691 10692 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:42.286152 10469 tablet.cc:1620] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:42.290897 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=155be183c0bf4a4ea3629f4db2c70747] alter complete (version 2)
I20240501 07:58:42.317490 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:42.322569 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:42.325125 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:42.327418 7121 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:42.328127 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:42.339179 9238 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:42.343498 9515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.348026 9535 raft_consensus.cc:1203] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.362839 9383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.395906 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:42.403674 10469 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:42.404104 10462 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:42.405841 10692 tablet.cc:1722] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:42.408818 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=155be183c0bf4a4ea3629f4db2c70747] alter complete (version 3)
I20240501 07:58:42.426406 7139 consensus_queue.cc:237] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:42.427721 7271 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.427731 9266 raft_consensus.cc:1261] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.428148 10346 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.428264 9942 consensus_queue.cc:1035] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.429999 10346 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:42.430644 9266 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:42.431743 7271 raft_consensus.cc:2932] T 00cac44dfd0f4d97a3056b590748ba58 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:42.433876 6737 catalog_manager.cc:5484] T 00cac44dfd0f4d97a3056b590748ba58 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:42.437003 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cac44dfd0f4d97a3056b590748ba58 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:42.437247 9864 tablet_replica.cc:330] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:42.437397 9864 raft_consensus.cc:2229] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.437553 9864 raft_consensus.cc:2258] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.437953 9864 ts_tablet_manager.cc:1875] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:42.439708 9864 ts_tablet_manager.cc:1888] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:42.439791 9864 log.cc:1196] T 00cac44dfd0f4d97a3056b590748ba58 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/00cac44dfd0f4d97a3056b590748ba58
I20240501 07:58:42.440333 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 00cac44dfd0f4d97a3056b590748ba58 (table rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]) successfully deleted
I20240501 07:58:42.449505 10636 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.453636 9877 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.469560 9734 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.473878 9882 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.510767 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:42.510931 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:42.512238 9665 raft_consensus.cc:1203] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.514168 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 8377c905ed6343bfa102b3aa9699e8e9
I20240501 07:58:42.514922 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:42.515089 9383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:42.515499 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 8377c905ed6343bfa102b3aa9699e8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:42.520401 10835 tablet_replica.cc:330] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:42.520577 10835 raft_consensus.cc:2229] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.519272 10833 tablet_replica.cc:330] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:42.520826 10833 raft_consensus.cc:2229] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.521143 10835 raft_consensus.cc:2258] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.521297 10833 raft_consensus.cc:2258] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.521908 10833 ts_tablet_manager.cc:1875] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.522518 10835 ts_tablet_manager.cc:1875] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.524996 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:42.529161 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:42.532513 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:42.536361 10253 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.542883 10832 tablet_replica.cc:330] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:42.543054 10832 raft_consensus.cc:2229] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.543181 10832 raft_consensus.cc:2258] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.548627 10747 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:42.551908 7121 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:42.556205 9515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.558755 9238 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.559113 9383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.564026 10835 ts_tablet_manager.cc:1888] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.527609 10833 ts_tablet_manager.cc:1888] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.566555 10833 log.cc:1196] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8377c905ed6343bfa102b3aa9699e8e9
I20240501 07:58:42.566902 10833 ts_tablet_manager.cc:1909] T 8377c905ed6343bfa102b3aa9699e8e9 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:42.566994 10835 log.cc:1196] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8377c905ed6343bfa102b3aa9699e8e9
I20240501 07:58:42.567391 10835 ts_tablet_manager.cc:1909] T 8377c905ed6343bfa102b3aa9699e8e9 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:42.568100 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table_new_name [id=155be183c0bf4a4ea3629f4db2c70747]) successfully deleted
I20240501 07:58:42.568720 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table_new_name [id=155be183c0bf4a4ea3629f4db2c70747]) successfully deleted
I20240501 07:58:42.571511 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:42.582396 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:42.590373 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 22cb0ec763ff4941acbfbb007995241d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.590725 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cb0ec763ff4941acbfbb007995241d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.591217 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 22cb0ec763ff4941acbfbb007995241d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.591311 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 22cb0ec763ff4941acbfbb007995241d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.591481 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cb0ec763ff4941acbfbb007995241d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.591609 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cb0ec763ff4941acbfbb007995241d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.592785 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 1037ad1dda6448f7a026fc685642d5c6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.593003 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037ad1dda6448f7a026fc685642d5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.593811 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 1037ad1dda6448f7a026fc685642d5c6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.594035 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037ad1dda6448f7a026fc685642d5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.595342 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 1037ad1dda6448f7a026fc685642d5c6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.595597 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037ad1dda6448f7a026fc685642d5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.596246 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2b974bc34cecbc7b2012c62ac245 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.596450 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2b974bc34cecbc7b2012c62ac245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.598321 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2b974bc34cecbc7b2012c62ac245 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.598596 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2b974bc34cecbc7b2012c62ac245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.600850 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 37ac2b974bc34cecbc7b2012c62ac245 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:42.601083 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ac2b974bc34cecbc7b2012c62ac245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.602566 10840 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:42.603277 10832 ts_tablet_manager.cc:1875] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.603526 10840 tablet_bootstrap.cc:654] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.605099 10843 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:42.605244 10844 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:42.606065 10832 ts_tablet_manager.cc:1888] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.606163 10832 log.cc:1196] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/8377c905ed6343bfa102b3aa9699e8e9
I20240501 07:58:42.606464 10832 ts_tablet_manager.cc:1909] T 8377c905ed6343bfa102b3aa9699e8e9 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:42.607147 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 8377c905ed6343bfa102b3aa9699e8e9 (table rebalancer_dynamic_table_new_name [id=155be183c0bf4a4ea3629f4db2c70747]) successfully deleted
I20240501 07:58:42.607676 10846 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:42.608098 10843 tablet_bootstrap.cc:654] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.608223 10844 tablet_bootstrap.cc:654] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.609010 10846 tablet_bootstrap.cc:654] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.610833 10844 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:42.610956 10844 ts_tablet_manager.cc:1367] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:42.611409 10844 raft_consensus.cc:348] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.611637 10844 raft_consensus.cc:374] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.611712 10844 raft_consensus.cc:729] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.611894 10844 consensus_queue.cc:260] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.612838 10844 ts_tablet_manager.cc:1398] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:42.613603 10843 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:42.613731 10843 ts_tablet_manager.cc:1367] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:42.615530 10842 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:42.616418 10843 raft_consensus.cc:348] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.616597 10842 tablet_bootstrap.cc:654] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.616595 10843 raft_consensus.cc:374] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.616641 10843 raft_consensus.cc:729] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.617008 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:42.616902 10841 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:42.619334 10841 tablet_bootstrap.cc:654] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.619683 10843 consensus_queue.cc:260] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.621587 10843 ts_tablet_manager.cc:1398] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:42.622551 10840 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:42.622655 10840 ts_tablet_manager.cc:1367] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:58:42.622814 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 57783d10ddfd4d1184a482e798b01a5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]), partition=
I20240501 07:58:42.623051 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57783d10ddfd4d1184a482e798b01a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.623113 10840 raft_consensus.cc:348] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.623229 10840 raft_consensus.cc:374] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.623272 10840 raft_consensus.cc:729] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.623412 10840 consensus_queue.cc:260] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.623662 10840 ts_tablet_manager.cc:1398] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:42.623826 10840 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:42.624624 10840 tablet_bootstrap.cc:654] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.624918 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 57783d10ddfd4d1184a482e798b01a5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]), partition=
I20240501 07:58:42.625118 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57783d10ddfd4d1184a482e798b01a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.625633 10840 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:42.625705 10840 ts_tablet_manager.cc:1367] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:42.626085 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 57783d10ddfd4d1184a482e798b01a5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]), partition=
I20240501 07:58:42.626313 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57783d10ddfd4d1184a482e798b01a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.626843 10840 raft_consensus.cc:348] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.626941 10840 raft_consensus.cc:374] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.627442 10840 raft_consensus.cc:729] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.627599 10840 consensus_queue.cc:260] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.627914 10840 ts_tablet_manager.cc:1398] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:42.628072 10840 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:42.628602 10846 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:42.628701 10846 ts_tablet_manager.cc:1367] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240501 07:58:42.629015 10840 tablet_bootstrap.cc:654] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.629173 10846 raft_consensus.cc:348] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.629284 10846 raft_consensus.cc:374] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.629323 10846 raft_consensus.cc:729] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.629474 10846 consensus_queue.cc:260] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.629791 10846 ts_tablet_manager.cc:1398] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:42.630509 10842 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:42.630595 10842 ts_tablet_manager.cc:1367] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:42.631124 10842 raft_consensus.cc:348] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.631235 10842 raft_consensus.cc:374] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.631276 10842 raft_consensus.cc:729] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.631472 10842 consensus_queue.cc:260] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.632107 10842 ts_tablet_manager.cc:1398] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:42.632257 10842 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:42.634056 10842 tablet_bootstrap.cc:654] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.635471 10841 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:42.635536 10842 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:42.635634 10841 ts_tablet_manager.cc:1367] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240501 07:58:42.635689 10842 ts_tablet_manager.cc:1367] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:42.636126 10841 raft_consensus.cc:348] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.636155 10842 raft_consensus.cc:348] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.636246 10842 raft_consensus.cc:374] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.636351 10842 raft_consensus.cc:729] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.636350 10841 raft_consensus.cc:374] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.636401 10841 raft_consensus.cc:729] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.636543 10841 consensus_queue.cc:260] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.636575 10842 consensus_queue.cc:260] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.636754 10843 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:42.637368 10842 ts_tablet_manager.cc:1398] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:42.637574 10842 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:42.637635 10841 ts_tablet_manager.cc:1398] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:42.639041 10842 tablet_bootstrap.cc:654] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.639551 10843 tablet_bootstrap.cc:654] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.640730 10853 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:42.641121 10843 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:42.641222 10843 ts_tablet_manager.cc:1367] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:42.641714 10843 raft_consensus.cc:348] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.641834 10843 raft_consensus.cc:374] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.641873 10843 raft_consensus.cc:729] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.642002 10843 consensus_queue.cc:260] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.642282 10843 ts_tablet_manager.cc:1398] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:42.643640 10706 raft_consensus.cc:482] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.643736 10706 raft_consensus.cc:504] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.644099 10853 tablet_bootstrap.cc:654] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:42.644173 10706 leader_election.cc:288] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:42.644621 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:42.644629 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:42.644754 6876 raft_consensus.cc:2454] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
W20240501 07:58:42.645100 6945 leader_election.cc:341] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:42.645749 6945 leader_election.cc:302] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters: fa6964452854415a85ad72f8639768d6
I20240501 07:58:42.646014 10706 raft_consensus.cc:2781] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.646095 10706 raft_consensus.cc:482] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.646153 10706 raft_consensus.cc:3037] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.646694 10853 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:42.646776 10853 ts_tablet_manager.cc:1367] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:42.647156 10853 raft_consensus.cc:348] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.647343 10853 raft_consensus.cc:374] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.647423 10853 raft_consensus.cc:729] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.647616 10853 consensus_queue.cc:260] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.647733 10706 raft_consensus.cc:504] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.648118 10706 leader_election.cc:288] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:42.648650 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:42.648797 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:42.648906 6876 raft_consensus.cc:3037] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.649672 10853 ts_tablet_manager.cc:1398] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:42.649950 6876 raft_consensus.cc:2454] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:42.650310 10840 tablet_bootstrap.cc:492] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:42.650377 6945 leader_election.cc:302] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:42.650436 10840 ts_tablet_manager.cc:1367] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
W20240501 07:58:42.650574 6945 leader_election.cc:341] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:42.650591 10706 raft_consensus.cc:2781] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.650712 10706 raft_consensus.cc:686] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:42.650826 10706 consensus_queue.cc:237] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.650933 10840 raft_consensus.cc:348] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.651058 10840 raft_consensus.cc:374] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.651110 10840 raft_consensus.cc:729] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.651799 10314 raft_consensus.cc:482] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.651932 10314 raft_consensus.cc:504] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.652408 10314 leader_election.cc:288] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:42.652546 10840 consensus_queue.cc:260] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.653038 10840 ts_tablet_manager.cc:1398] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:42.653188 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cb0ec763ff4941acbfbb007995241d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:42.653302 9535 raft_consensus.cc:2454] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:42.653748 7208 leader_election.cc:302] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:42.654016 6737 catalog_manager.cc:5484] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:42.654749 10314 raft_consensus.cc:2781] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.654811 10314 raft_consensus.cc:482] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.654817 10842 tablet_bootstrap.cc:492] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:42.654861 10314 raft_consensus.cc:3037] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.654891 10842 ts_tablet_manager.cc:1367] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.001s
I20240501 07:58:42.655880 10842 raft_consensus.cc:348] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.655882 10314 raft_consensus.cc:504] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.655975 10842 raft_consensus.cc:374] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:42.656004 10842 raft_consensus.cc:729] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:42.656123 10842 consensus_queue.cc:260] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.656280 10314 leader_election.cc:288] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:42.656412 10842 ts_tablet_manager.cc:1398] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:42.657055 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cb0ec763ff4941acbfbb007995241d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:42.657176 9535 raft_consensus.cc:3037] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.658008 9535 raft_consensus.cc:2454] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:42.658279 7208 leader_election.cc:302] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:42.658704 10041 raft_consensus.cc:2781] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.658787 10041 raft_consensus.cc:686] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:42.658926 10041 consensus_queue.cc:237] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.661653 6733 catalog_manager.cc:5484] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:42.662803 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cb0ec763ff4941acbfbb007995241d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:42.662928 9403 raft_consensus.cc:2454] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:42.663144 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cb0ec763ff4941acbfbb007995241d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:42.663259 9402 raft_consensus.cc:3037] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.664264 9402 raft_consensus.cc:2454] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:42.669086 7141 consensus_queue.cc:237] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:42.670326 9535 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.670441 6876 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.670811 9942 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.670975 10346 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.672870 10777 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:42.673278 6876 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:42.673465 9535 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:42.676533 6733 catalog_manager.cc:5484] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:42.679220 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet f334b66a84b94c3ba15004253b4cb714 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.679622 10303 tablet_replica.cc:330] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.679797 10303 raft_consensus.cc:2229] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.679944 10303 raft_consensus.cc:2258] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.680449 10303 ts_tablet_manager.cc:1875] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:42.682505 10303 ts_tablet_manager.cc:1888] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:42.682559 10303 log.cc:1196] T f334b66a84b94c3ba15004253b4cb714 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f334b66a84b94c3ba15004253b4cb714
I20240501 07:58:42.682979 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet f334b66a84b94c3ba15004253b4cb714 (table rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]) successfully deleted
I20240501 07:58:42.694605 9735 raft_consensus.cc:482] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.694721 9735 raft_consensus.cc:504] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.695245 9735 leader_election.cc:288] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.698719 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:42.698776 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:42.698874 9665 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:42.698895 9402 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:42.699347 9470 leader_election.cc:302] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:42.699877 9735 raft_consensus.cc:2781] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.699934 9735 raft_consensus.cc:482] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.699971 9735 raft_consensus.cc:3037] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.700994 9735 raft_consensus.cc:504] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.701275 9735 leader_election.cc:288] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:42.701735 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:42.701855 9402 raft_consensus.cc:3037] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.702781 9402 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:42.703045 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:42.703130 9470 leader_election.cc:302] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:42.703176 9665 raft_consensus.cc:3037] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.703301 9735 raft_consensus.cc:2781] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.703485 9735 raft_consensus.cc:686] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:42.703627 9735 consensus_queue.cc:237] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:42.704061 9665 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:42.705543 6735 catalog_manager.cc:5484] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:42.752028 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 07:58:42.752179 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 07:58:42.754384 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet a42d4784867744b6be9e343ec6fdedbc
I20240501 07:58:42.754917 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet a42d4784867744b6be9e343ec6fdedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:42.754995 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet a42d4784867744b6be9e343ec6fdedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.755187 10303 tablet_replica.cc:330] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.755194 9864 tablet_replica.cc:330] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:42.755338 9864 raft_consensus.cc:2229] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.755338 10303 raft_consensus.cc:2229] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.755311 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet a42d4784867744b6be9e343ec6fdedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.755431 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet 37e73deb03274bba82ceb67087208bdd
I20240501 07:58:42.755457 9864 raft_consensus.cc:2258] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.755497 9868 tablet_replica.cc:330] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.755525 10303 raft_consensus.cc:2258] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.755616 9868 raft_consensus.cc:2229] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.755766 9868 raft_consensus.cc:2258] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.756001 10303 ts_tablet_manager.cc:1875] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.756001 9864 ts_tablet_manager.cc:1875] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.756222 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:42.756285 9868 ts_tablet_manager.cc:1875] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.756749 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e73deb03274bba82ceb67087208bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:42.757192 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e73deb03274bba82ceb67087208bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.757210 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:42.758044 10303 ts_tablet_manager.cc:1888] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.758044 9864 ts_tablet_manager.cc:1888] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.758109 10303 log.cc:1196] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a42d4784867744b6be9e343ec6fdedbc
I20240501 07:58:42.758126 9864 log.cc:1196] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a42d4784867744b6be9e343ec6fdedbc
I20240501 07:58:42.758390 9864 ts_tablet_manager.cc:1909] T a42d4784867744b6be9e343ec6fdedbc P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:42.758399 10303 ts_tablet_manager.cc:1909] T a42d4784867744b6be9e343ec6fdedbc P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:42.758596 10835 tablet_replica.cc:330] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:42.758612 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:42.758751 10835 raft_consensus.cc:2229] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.758857 10835 raft_consensus.cc:2258] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.759445 9868 ts_tablet_manager.cc:1888] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:42.759516 9868 log.cc:1196] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a42d4784867744b6be9e343ec6fdedbc
I20240501 07:58:42.759510 10835 ts_tablet_manager.cc:1875] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.759537 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e73deb03274bba82ceb67087208bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.760175 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e73deb03274bba82ceb67087208bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:42.760512 9868 ts_tablet_manager.cc:1909] T a42d4784867744b6be9e343ec6fdedbc P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.760689 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet a42d4784867744b6be9e343ec6fdedbc (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.760780 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:42.760876 10303 tablet_replica.cc:330] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.760960 10302 tablet_replica.cc:330] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:42.760990 10303 raft_consensus.cc:2229] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.761098 9868 tablet_replica.cc:330] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.761098 10302 raft_consensus.cc:2229] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.761117 10303 raft_consensus.cc:2258] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.761219 9868 raft_consensus.cc:2229] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.761359 10302 raft_consensus.cc:2258] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.761399 9868 raft_consensus.cc:2258] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.761610 10303 ts_tablet_manager.cc:1875] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.761626 10835 ts_tablet_manager.cc:1888] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.761694 10835 log.cc:1196] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/37e73deb03274bba82ceb67087208bdd
I20240501 07:58:42.761922 9868 ts_tablet_manager.cc:1875] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.761920 10302 ts_tablet_manager.cc:1875] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.761979 10835 ts_tablet_manager.cc:1909] T 37e73deb03274bba82ceb67087208bdd P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:42.762518 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b3a231d67e4092a94d3539e31e62f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:42 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:42.763348 9864 tablet_replica.cc:330] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:42.763437 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet a42d4784867744b6be9e343ec6fdedbc (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.763579 9864 raft_consensus.cc:2229] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.763825 9864 raft_consensus.cc:2258] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.763901 10303 ts_tablet_manager.cc:1888] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.764004 10303 log.cc:1196] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/37e73deb03274bba82ceb67087208bdd
I20240501 07:58:42.764024 9868 ts_tablet_manager.cc:1888] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.764024 10302 ts_tablet_manager.cc:1888] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.764106 10302 log.cc:1196] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/37e73deb03274bba82ceb67087208bdd
I20240501 07:58:42.764230 9868 log.cc:1196] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/37e73deb03274bba82ceb67087208bdd
I20240501 07:58:42.764358 10303 ts_tablet_manager.cc:1909] T 37e73deb03274bba82ceb67087208bdd P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:42.764433 10302 ts_tablet_manager.cc:1909] T 37e73deb03274bba82ceb67087208bdd P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:42.764429 9864 ts_tablet_manager.cc:1875] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.764521 9868 ts_tablet_manager.cc:1909] T 37e73deb03274bba82ceb67087208bdd P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.765017 9868 tablet_replica.cc:330] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:42.765143 9868 raft_consensus.cc:2229] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.765273 9868 raft_consensus.cc:2258] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.765712 9868 ts_tablet_manager.cc:1875] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.765826 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 37e73deb03274bba82ceb67087208bdd (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.766144 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 37e73deb03274bba82ceb67087208bdd (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.766463 9864 ts_tablet_manager.cc:1888] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.766474 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 37e73deb03274bba82ceb67087208bdd (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.766592 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet a42d4784867744b6be9e343ec6fdedbc (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.766698 10303 tablet_replica.cc:330] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:42.766789 9864 log.cc:1196] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:42.766795 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 37e73deb03274bba82ceb67087208bdd (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.766842 10303 raft_consensus.cc:2229] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:42.767089 10303 raft_consensus.cc:2258] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.767644 9864 ts_tablet_manager.cc:1909] T f4b3a231d67e4092a94d3539e31e62f4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:42.767861 9868 ts_tablet_manager.cc:1888] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.767989 9868 log.cc:1196] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:42.768074 10303 ts_tablet_manager.cc:1875] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.768326 9868 ts_tablet_manager.cc:1909] T f4b3a231d67e4092a94d3539e31e62f4 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:42.768440 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet f4b3a231d67e4092a94d3539e31e62f4 (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.768702 10923 tablet_replica.cc:330] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:42.769039 10923 raft_consensus.cc:2229] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:42.769119 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet f4b3a231d67e4092a94d3539e31e62f4 (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.769337 10923 raft_consensus.cc:2258] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:42.770022 10303 ts_tablet_manager.cc:1888] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.770112 10303 log.cc:1196] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:42.770296 10923 ts_tablet_manager.cc:1875] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:42.770337 10303 ts_tablet_manager.cc:1909] T f4b3a231d67e4092a94d3539e31e62f4 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:42.770946 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet f4b3a231d67e4092a94d3539e31e62f4 (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.772141 10923 ts_tablet_manager.cc:1888] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:42.772209 10923 log.cc:1196] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f4b3a231d67e4092a94d3539e31e62f4
I20240501 07:58:42.772392 10923 ts_tablet_manager.cc:1909] T f4b3a231d67e4092a94d3539e31e62f4 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:42.773284 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet f4b3a231d67e4092a94d3539e31e62f4 (table rebalance_test_table_10 [id=e658efc8bfca4c699ff1c5680ba5381c]) successfully deleted
I20240501 07:58:42.908177 10636 raft_consensus.cc:482] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.908329 10636 raft_consensus.cc:504] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.908788 10636 leader_election.cc:288] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:42.909169 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57783d10ddfd4d1184a482e798b01a5b" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:42.909188 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57783d10ddfd4d1184a482e798b01a5b" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:42.909303 9535 raft_consensus.cc:2454] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:42.909332 9271 raft_consensus.cc:2454] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:42.909595 9602 leader_election.cc:302] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:42.909749 10636 raft_consensus.cc:2781] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:42.909803 10636 raft_consensus.cc:482] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:42.909854 10636 raft_consensus.cc:3037] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.910656 10636 raft_consensus.cc:504] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.911020 10636 leader_election.cc:288] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:42.911302 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57783d10ddfd4d1184a482e798b01a5b" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:42.911348 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57783d10ddfd4d1184a482e798b01a5b" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:42.911442 9535 raft_consensus.cc:3037] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.911454 9271 raft_consensus.cc:3037] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.912209 9271 raft_consensus.cc:2454] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:42.912217 9535 raft_consensus.cc:2454] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:42.912482 9601 leader_election.cc:302] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:42.912711 10636 raft_consensus.cc:2781] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:42.912807 10636 raft_consensus.cc:686] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:42.912948 10636 consensus_queue.cc:237] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:42.914737 6732 catalog_manager.cc:5484] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:42.935299 7250 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:42.941258 9238 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.943639 6989 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:42.946462 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:42.953439 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.954423 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:42.955461 9507 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:42.969777 9382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:42.972142 9734 raft_consensus.cc:482] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:42.972242 9734 raft_consensus.cc:504] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:42.972667 9734 leader_election.cc:288] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:42.973263 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:42.973430 6876 raft_consensus.cc:2379] T 37ac2b974bc34cecbc7b2012c62ac245 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa6964452854415a85ad72f8639768d6 in current term 1: Already voted for candidate 9a3888360d4d4ace96b6b909c160bfc2 in this term.
I20240501 07:58:42.973912 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ac2b974bc34cecbc7b2012c62ac245" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:42.974277 9339 leader_election.cc:302] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:58:42.974536 9734 raft_consensus.cc:3037] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:42.995838 9734 raft_consensus.cc:2726] T 37ac2b974bc34cecbc7b2012c62ac245 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:43.007751 7009 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2"
dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34316
I20240501 07:58:43.007925 7009 raft_consensus.cc:595] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:43.017062 9936 raft_consensus.cc:979] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: : Instructing follower 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:58:43.017177 9936 raft_consensus.cc:1067] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Signalling peer 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:58:43.017709 7271 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2"
dest_uuid: "15ab96f6477b4291932409b80af07213"
from {username='slave'} at 127.0.99.130:45323
I20240501 07:58:43.017861 7271 raft_consensus.cc:482] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.017917 7271 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.018966 7271 raft_consensus.cc:504] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.019382 7271 leader_election.cc:288] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:43.019747 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:43.019858 6876 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.020066 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:43.020175 9535 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.020817 6876 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:43.021055 9535 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:43.021718 7208 leader_election.cc:302] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.021878 10041 raft_consensus.cc:2781] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.021941 10041 raft_consensus.cc:686] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:43.022061 10041 consensus_queue.cc:237] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.023903 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e711b7a9684137969ce2bbffae40c2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:43.024016 7009 raft_consensus.cc:3032] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.024058 7009 raft_consensus.cc:729] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:43.024116 6732 catalog_manager.cc:5484] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:43.024204 7009 consensus_queue.cc:260] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.024391 7009 raft_consensus.cc:3037] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.025308 7009 raft_consensus.cc:2454] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:43.033524 10706 consensus_queue.cc:1035] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.037609 9936 consensus_queue.cc:1035] T 37ac2b974bc34cecbc7b2012c62ac245 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.038296 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:43.049351 9271 raft_consensus.cc:1261] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.049451 9535 raft_consensus.cc:1261] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.049831 9877 consensus_queue.cc:1035] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.049957 10636 consensus_queue.cc:1035] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.053378 6734 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]). Expected version 1 got 0
I20240501 07:58:43.053726 6736 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]). Expected version 1 got 0
I20240501 07:58:43.054570 10957 mvcc.cc:205] Tried to move back new op lower bound from 7022798123202392064 to 7022798122652708864. Current Snapshot: MvccSnapshot[applied={T|T < 7022798123202392064}]
I20240501 07:58:43.054883 10966 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.055701 10966 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:43.056916 6732 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]). Expected version 1 got 0
I20240501 07:58:43.057473 6736 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff]). Expected version 1 got 0
I20240501 07:58:43.057968 10968 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.058311 10968 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:43.057967 10967 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.059980 10967 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:43.063470 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff] alter complete (version 1)
I20240501 07:58:43.092015 9735 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.100636 9238 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.102244 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:43.104560 10891 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.114346 7250 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:43.114957 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.115727 9382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.116550 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:43.117812 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.124004 10041 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.144440 9507 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:43.156875 10314 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.168797 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:43.174340 10966 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.174714 10966 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:43.175784 10968 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.176092 10968 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:43.176143 10967 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.176443 10967 tablet.cc:1620] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:43.181409 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c30614faa1154172b0409dc0f5a9d4ff] alter complete (version 2)
I20240501 07:58:43.219836 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:43.226879 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4ea1beadd542a894f002ae78d3b37f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.226876 9507 tablet_service.cc:1467] Processing CreateTablet for tablet f8921a8351e04daba09a8e7e4f683fd9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.227218 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8921a8351e04daba09a8e7e4f683fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.227219 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4ea1beadd542a894f002ae78d3b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.229542 11045 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:43.230425 9238 tablet_service.cc:1467] Processing CreateTablet for tablet f8921a8351e04daba09a8e7e4f683fd9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.230738 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8921a8351e04daba09a8e7e4f683fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.233089 9251 tablet_service.cc:1467] Processing CreateTablet for tablet b4c8812331c847c9941e52daf0313897 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.233258 11046 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:43.233297 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8812331c847c9941e52daf0313897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.234063 11046 tablet_bootstrap.cc:654] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.234197 9645 tablet_service.cc:1467] Processing CreateTablet for tablet f8921a8351e04daba09a8e7e4f683fd9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.234465 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8921a8351e04daba09a8e7e4f683fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.234581 11045 tablet_bootstrap.cc:654] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.235337 11046 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:43.235404 11046 ts_tablet_manager.cc:1367] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.235778 11046 raft_consensus.cc:348] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.235909 11046 raft_consensus.cc:374] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.235945 11046 raft_consensus.cc:729] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.235936 9382 tablet_service.cc:1467] Processing CreateTablet for tablet b4c8812331c847c9941e52daf0313897 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.236057 11046 consensus_queue.cc:260] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.236196 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8812331c847c9941e52daf0313897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.236317 11046 ts_tablet_manager.cc:1398] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:43.236550 11050 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:43.236680 9646 tablet_service.cc:1467] Processing CreateTablet for tablet b4c8812331c847c9941e52daf0313897 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.236764 11045 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:43.236841 11045 ts_tablet_manager.cc:1367] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:43.236888 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8812331c847c9941e52daf0313897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.237272 11045 raft_consensus.cc:348] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.237370 11045 raft_consensus.cc:374] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.237421 11045 raft_consensus.cc:729] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.237495 11050 tablet_bootstrap.cc:654] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.237596 11045 consensus_queue.cc:260] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.238396 11052 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:43.238472 11045 ts_tablet_manager.cc:1398] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.238610 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4ea1beadd542a894f002ae78d3b37f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.238832 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4ea1beadd542a894f002ae78d3b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.238895 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4ea1beadd542a894f002ae78d3b37f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.239126 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4ea1beadd542a894f002ae78d3b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.239881 11052 tablet_bootstrap.cc:654] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.242033 11047 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.242883 11047 tablet_bootstrap.cc:654] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.244673 11047 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.244808 11047 ts_tablet_manager.cc:1367] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:43.245841 11047 raft_consensus.cc:348] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.245955 11047 raft_consensus.cc:374] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.245990 11047 raft_consensus.cc:729] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.246151 11047 consensus_queue.cc:260] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.246582 11047 ts_tablet_manager.cc:1398] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.246770 11047 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.247509 11052 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:43.247589 11052 ts_tablet_manager.cc:1367] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:58:43.247889 11047 tablet_bootstrap.cc:654] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.247993 11052 raft_consensus.cc:348] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.248087 11052 raft_consensus.cc:374] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.248111 11052 raft_consensus.cc:729] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.248257 11052 consensus_queue.cc:260] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.248531 11052 ts_tablet_manager.cc:1398] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:43.248725 11052 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:43.248934 11050 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:43.249014 11050 ts_tablet_manager.cc:1367] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:43.249183 11047 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.249251 11047 ts_tablet_manager.cc:1367] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:43.249471 11052 tablet_bootstrap.cc:654] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.249444 11050 raft_consensus.cc:348] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.249529 11050 raft_consensus.cc:374] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.249559 11050 raft_consensus.cc:729] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.249626 11047 raft_consensus.cc:348] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.249702 11047 raft_consensus.cc:374] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.249679 11050 consensus_queue.cc:260] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.249728 11047 raft_consensus.cc:729] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.249958 11050 ts_tablet_manager.cc:1398] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.249853 11047 consensus_queue.cc:260] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.250141 11050 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:43.250458 11047 ts_tablet_manager.cc:1398] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.251495 11052 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:43.251577 11052 ts_tablet_manager.cc:1367] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:43.252027 11052 raft_consensus.cc:348] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.252120 11052 raft_consensus.cc:374] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.252175 11052 raft_consensus.cc:729] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.252287 11052 consensus_queue.cc:260] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.252511 11052 ts_tablet_manager.cc:1398] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:43.255165 11050 tablet_bootstrap.cc:654] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.256242 11050 tablet_bootstrap.cc:492] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:43.256327 11050 ts_tablet_manager.cc:1367] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:58:43.256693 9734 raft_consensus.cc:482] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.256783 9734 raft_consensus.cc:504] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.257159 9734 leader_election.cc:288] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.257488 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8812331c847c9941e52daf0313897" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:43.257511 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8812331c847c9941e52daf0313897" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:43.257601 9271 raft_consensus.cc:2454] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:43.257807 11050 raft_consensus.cc:348] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.257831 9338 leader_election.cc:302] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:43.257891 11050 raft_consensus.cc:374] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.257921 11050 raft_consensus.cc:729] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
W20240501 07:58:43.258018 9338 leader_election.cc:341] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:43.258031 9734 raft_consensus.cc:2781] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.258100 9734 raft_consensus.cc:482] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.258071 11050 consensus_queue.cc:260] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.258143 9734 raft_consensus.cc:3037] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.258534 11050 ts_tablet_manager.cc:1398] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:43.258718 11050 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:43.259264 9734 raft_consensus.cc:504] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.259613 9734 leader_election.cc:288] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.259666 11050 tablet_bootstrap.cc:654] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.259954 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8812331c847c9941e52daf0313897" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:43.260073 9271 raft_consensus.cc:3037] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.260131 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8812331c847c9941e52daf0313897" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:43.260213 9666 raft_consensus.cc:3037] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.261052 9271 raft_consensus.cc:2454] T b4c8812331c847c9941e52daf0313897 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:43.261119 9666 raft_consensus.cc:2454] T b4c8812331c847c9941e52daf0313897 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:43.261178 11050 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:43.261256 11050 ts_tablet_manager.cc:1367] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:43.261400 9338 leader_election.cc:302] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:43.261541 9734 raft_consensus.cc:2781] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.261673 9734 raft_consensus.cc:686] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:43.261744 11050 raft_consensus.cc:348] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.261829 11050 raft_consensus.cc:374] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.261858 11050 raft_consensus.cc:729] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.261796 9734 consensus_queue.cc:237] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:43.263510 11050 consensus_queue.cc:260] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.263926 11050 ts_tablet_manager.cc:1398] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:43.264904 6729 catalog_manager.cc:5484] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:43.267611 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 07:58:43.267719 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 07:58:43.270812 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 14165b2b100b4f818ccececc54bd8bad
I20240501 07:58:43.271987 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet 33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:43.272243 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 00569cfef8704ea58ddfeb7bc7c0e20d
I20240501 07:58:43.272939 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e711b7a9684137969ce2bbffae40c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:43.272982 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 14165b2b100b4f818ccececc54bd8bad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.273175 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e711b7a9684137969ce2bbffae40c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.273175 6988 tablet_service.cc:1514] Processing DeleteTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.273353 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 14165b2b100b4f818ccececc54bd8bad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:43.273408 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:43.273456 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 00569cfef8704ea58ddfeb7bc7c0e20d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:43.273484 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e711b7a9684137969ce2bbffae40c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:43.274207 11081 tablet_replica.cc:330] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:43.274389 11081 raft_consensus.cc:2229] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.274554 11081 raft_consensus.cc:2258] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.274796 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e711b7a9684137969ce2bbffae40c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:43.275768 11081 ts_tablet_manager.cc:1875] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.276612 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 14165b2b100b4f818ccececc54bd8bad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:43.277480 11079 tablet_replica.cc:330] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.277635 11079 raft_consensus.cc:2229] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.277849 11079 raft_consensus.cc:2258] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.278230 11081 ts_tablet_manager.cc:1888] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.278401 11079 ts_tablet_manager.cc:1875] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.278494 11081 log.cc:1196] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/00569cfef8704ea58ddfeb7bc7c0e20d
I20240501 07:58:43.278774 11081 ts_tablet_manager.cc:1909] T 00569cfef8704ea58ddfeb7bc7c0e20d P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:43.279434 11081 tablet_replica.cc:330] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:43.279553 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 00569cfef8704ea58ddfeb7bc7c0e20d (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.279667 11081 raft_consensus.cc:2229] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.279877 11081 pending_rounds.cc:62] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Trying to abort 1 pending ops.
I20240501 07:58:43.279994 11080 tablet_replica.cc:330] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:43.279999 11081 pending_rounds.cc:68] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Aborting op as it isn't in flight: id { term: 2 index: 5 } timestamp: 7022798123099901952 op_type: NO_OP noop_request { }
I20240501 07:58:43.280092 11081 raft_consensus.cc:2866] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 07:58:43.280151 11080 raft_consensus.cc:2229] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.280193 11081 raft_consensus.cc:2258] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.280303 11080 raft_consensus.cc:2258] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.280737 11081 ts_tablet_manager.cc:1875] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.280829 11080 ts_tablet_manager.cc:1875] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.281276 11078 tablet_replica.cc:330] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:43.281451 11078 raft_consensus.cc:2229] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.281608 11078 raft_consensus.cc:2258] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.282227 11078 ts_tablet_manager.cc:1875] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.282850 11081 ts_tablet_manager.cc:1888] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20240501 07:58:43.282966 11081 log.cc:1196] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:43.283321 11081 ts_tablet_manager.cc:1909] T 33e711b7a9684137969ce2bbffae40c2 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:43.284158 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 33e711b7a9684137969ce2bbffae40c2 (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.284792 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:43.288839 11082 tablet_replica.cc:330] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:43.289054 11082 raft_consensus.cc:2229] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.289322 11082 raft_consensus.cc:2258] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.290903 11082 ts_tablet_manager.cc:1875] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.291787 10966 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.292310 10967 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.293525 10968 tablet.cc:1722] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.296989 11079 ts_tablet_manager.cc:1888] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.297113 11079 log.cc:1196] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/14165b2b100b4f818ccececc54bd8bad
I20240501 07:58:43.297549 11078 ts_tablet_manager.cc:1888] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.297619 11078 log.cc:1196] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:43.298158 11079 ts_tablet_manager.cc:1909] T 14165b2b100b4f818ccececc54bd8bad P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.298564 11078 ts_tablet_manager.cc:1909] T 33e711b7a9684137969ce2bbffae40c2 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:43.298754 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=c30614faa1154172b0409dc0f5a9d4ff] alter complete (version 3)
I20240501 07:58:43.299429 11079 tablet_replica.cc:330] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.299476 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 14165b2b100b4f818ccececc54bd8bad (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.299590 11079 raft_consensus.cc:2229] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.299683 11079 raft_consensus.cc:2258] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.300132 11079 ts_tablet_manager.cc:1875] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.300309 11080 ts_tablet_manager.cc:1888] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.300438 11080 log.cc:1196] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/14165b2b100b4f818ccececc54bd8bad
I20240501 07:58:43.300808 11080 ts_tablet_manager.cc:1909] T 14165b2b100b4f818ccececc54bd8bad P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:43.301481 11080 tablet_replica.cc:330] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:43.301560 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 14165b2b100b4f818ccececc54bd8bad (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.301697 11080 raft_consensus.cc:2229] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.301851 11080 raft_consensus.cc:2258] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.302157 11079 ts_tablet_manager.cc:1888] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.302218 11079 log.cc:1196] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:43.302420 11080 ts_tablet_manager.cc:1875] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.302460 11079 ts_tablet_manager.cc:1909] T 33e711b7a9684137969ce2bbffae40c2 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.302951 11079 tablet_replica.cc:330] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.302997 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 33e711b7a9684137969ce2bbffae40c2 (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.303418 11079 raft_consensus.cc:2229] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.303411 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 33e711b7a9684137969ce2bbffae40c2 (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.303601 11079 raft_consensus.cc:2258] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.304000 11079 ts_tablet_manager.cc:1875] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.304241 11082 ts_tablet_manager.cc:1888] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.304396 11082 log.cc:1196] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/33e711b7a9684137969ce2bbffae40c2
I20240501 07:58:43.304620 11080 ts_tablet_manager.cc:1888] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.304700 11080 log.cc:1196] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/00569cfef8704ea58ddfeb7bc7c0e20d
I20240501 07:58:43.304740 11082 ts_tablet_manager.cc:1909] T 33e711b7a9684137969ce2bbffae40c2 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:43.304986 11080 ts_tablet_manager.cc:1909] T 00569cfef8704ea58ddfeb7bc7c0e20d P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:43.305513 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 33e711b7a9684137969ce2bbffae40c2 (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.305828 11079 ts_tablet_manager.cc:1888] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.305886 11082 tablet_replica.cc:330] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:43.305904 11079 log.cc:1196] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/00569cfef8704ea58ddfeb7bc7c0e20d
I20240501 07:58:43.306041 11082 raft_consensus.cc:2229] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.306141 11079 ts_tablet_manager.cc:1909] T 00569cfef8704ea58ddfeb7bc7c0e20d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.306169 11082 raft_consensus.cc:2258] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.306296 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 00569cfef8704ea58ddfeb7bc7c0e20d (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.306694 11082 ts_tablet_manager.cc:1875] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.306687 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 00569cfef8704ea58ddfeb7bc7c0e20d (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.308148 11082 ts_tablet_manager.cc:1888] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.308212 11082 log.cc:1196] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/14165b2b100b4f818ccececc54bd8bad
I20240501 07:58:43.308424 11082 ts_tablet_manager.cc:1909] T 14165b2b100b4f818ccececc54bd8bad P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:43.309019 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 14165b2b100b4f818ccececc54bd8bad (table rebalance_test_table_12 [id=0b8dc1a66bc74b4fa1aba2497ffe6e43]) successfully deleted
I20240501 07:58:43.347193 10891 raft_consensus.cc:482] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.347337 10041 raft_consensus.cc:482] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.347357 10891 raft_consensus.cc:504] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.347438 10041 raft_consensus.cc:504] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.347760 10891 leader_election.cc:288] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.347790 10041 leader_election.cc:288] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.348373 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:43.348402 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:43.348532 9271 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.348520 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4ea1beadd542a894f002ae78d3b37f" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:43.348630 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4ea1beadd542a894f002ae78d3b37f" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:43.348651 9666 raft_consensus.cc:2454] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:43.348733 9402 raft_consensus.cc:2454] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:43.348524 9665 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.348913 9470 leader_election.cc:302] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.349360 10891 raft_consensus.cc:2781] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.349431 10891 raft_consensus.cc:482] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.349479 10891 raft_consensus.cc:3037] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.350548 10891 raft_consensus.cc:504] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.350885 10891 leader_election.cc:288] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.351315 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:43.351336 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:43.351351 7207 leader_election.cc:302] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:43.351429 9665 raft_consensus.cc:3037] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.351433 9271 raft_consensus.cc:3037] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.351696 10041 raft_consensus.cc:2781] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.351749 10041 raft_consensus.cc:482] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.351787 10041 raft_consensus.cc:3037] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.352316 9665 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:43.352453 9271 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:43.352640 9470 leader_election.cc:302] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.352800 10041 raft_consensus.cc:504] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.353235 10041 leader_election.cc:288] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.353475 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4ea1beadd542a894f002ae78d3b37f" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:43.353574 9402 raft_consensus.cc:3037] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.353605 10891 raft_consensus.cc:2781] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.353634 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4ea1beadd542a894f002ae78d3b37f" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:43.353695 10891 raft_consensus.cc:686] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:43.353734 9665 raft_consensus.cc:3037] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.353852 10891 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:43.354430 9402 raft_consensus.cc:2454] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:43.354584 9665 raft_consensus.cc:2454] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:43.355383 7207 leader_election.cc:302] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:43.355708 10041 raft_consensus.cc:2781] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.355804 10041 raft_consensus.cc:686] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:43.355935 10041 consensus_queue.cc:237] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.356133 6731 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:43.358228 6736 catalog_manager.cc:5484] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:43.380257 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:43.384955 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:43.392419 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.394713 9251 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.394856 6988 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.401000 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:43.402472 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:43.402617 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:43.406579 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 57783d10ddfd4d1184a482e798b01a5b
I20240501 07:58:43.407178 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 57783d10ddfd4d1184a482e798b01a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:43.407371 11078 tablet_replica.cc:330] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:43.407496 11078 raft_consensus.cc:2229] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.407562 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet 57783d10ddfd4d1184a482e798b01a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:43.407619 11078 raft_consensus.cc:2258] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.408097 9507 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:43.408234 11078 ts_tablet_manager.cc:1875] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.408506 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 57783d10ddfd4d1184a482e798b01a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:43.410494 11078 ts_tablet_manager.cc:1888] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.410565 11078 log.cc:1196] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/57783d10ddfd4d1184a482e798b01a5b
I20240501 07:58:43.410821 11078 ts_tablet_manager.cc:1909] T 57783d10ddfd4d1184a482e798b01a5b P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:43.411502 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table_new_name [id=c30614faa1154172b0409dc0f5a9d4ff]) successfully deleted
I20240501 07:58:43.425566 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.429431 11097 tablet_replica.cc:330] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:43.429646 11097 raft_consensus.cc:2229] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.431174 11097 raft_consensus.cc:2258] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.434521 11097 ts_tablet_manager.cc:1875] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.440727 11097 ts_tablet_manager.cc:1888] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.440799 11097 log.cc:1196] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/57783d10ddfd4d1184a482e798b01a5b
I20240501 07:58:43.443337 11097 ts_tablet_manager.cc:1909] T 57783d10ddfd4d1184a482e798b01a5b P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:43.444105 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table_new_name [id=c30614faa1154172b0409dc0f5a9d4ff]) successfully deleted
I20240501 07:58:43.450325 11096 tablet_replica.cc:330] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:43.450486 11096 raft_consensus.cc:2229] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.450627 11096 raft_consensus.cc:2258] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.451295 11096 ts_tablet_manager.cc:1875] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.453684 11096 ts_tablet_manager.cc:1888] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.453827 11096 log.cc:1196] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/57783d10ddfd4d1184a482e798b01a5b
I20240501 07:58:43.455039 11096 ts_tablet_manager.cc:1909] T 57783d10ddfd4d1184a482e798b01a5b P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:43.455789 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 57783d10ddfd4d1184a482e798b01a5b (table rebalancer_dynamic_table_new_name [id=c30614faa1154172b0409dc0f5a9d4ff]) successfully deleted
I20240501 07:58:43.480346 7271 consensus_queue.cc:237] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.481644 9271 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.481829 7140 raft_consensus.cc:1261] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.482017 10041 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.482251 10314 consensus_queue.cc:1035] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.483650 10041 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.484027 9271 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.486313 7140 raft_consensus.cc:2932] T 1671c966902c4eb98b238c4419c525fe P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.486647 6736 catalog_manager.cc:5484] T 1671c966902c4eb98b238c4419c525fe P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.489897 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 1671c966902c4eb98b238c4419c525fe with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:43.490664 11082 tablet_replica.cc:330] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:43.490818 11082 raft_consensus.cc:2229] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.490968 11082 raft_consensus.cc:2258] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.491432 11082 ts_tablet_manager.cc:1875] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.493150 11082 ts_tablet_manager.cc:1888] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:43.493233 11082 log.cc:1196] T 1671c966902c4eb98b238c4419c525fe P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1671c966902c4eb98b238c4419c525fe
I20240501 07:58:43.493736 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 1671c966902c4eb98b238c4419c525fe (table rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]) successfully deleted
I20240501 07:58:43.508128 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:43.512729 9507 tablet_service.cc:1467] Processing CreateTablet for tablet a1952714471f4771b28b78fe3b9a8c6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]), partition=
I20240501 07:58:43.512985 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1952714471f4771b28b78fe3b9a8c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.515107 11045 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:43.515275 9251 tablet_service.cc:1467] Processing CreateTablet for tablet a1952714471f4771b28b78fe3b9a8c6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]), partition=
I20240501 07:58:43.515493 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1952714471f4771b28b78fe3b9a8c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.515957 11045 tablet_bootstrap.cc:654] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.516733 7251 tablet_service.cc:1467] Processing CreateTablet for tablet a1952714471f4771b28b78fe3b9a8c6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]), partition=
I20240501 07:58:43.516980 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1952714471f4771b28b78fe3b9a8c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.517153 11045 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:43.517225 11045 ts_tablet_manager.cc:1367] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.517849 11045 raft_consensus.cc:348] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.517918 11047 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.517956 11045 raft_consensus.cc:374] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.517987 11045 raft_consensus.cc:729] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.518092 11045 consensus_queue.cc:260] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.518404 11045 ts_tablet_manager.cc:1398] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.518767 11047 tablet_bootstrap.cc:654] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.519001 11046 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:43.519804 11046 tablet_bootstrap.cc:654] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.520282 11047 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.520359 11047 ts_tablet_manager.cc:1367] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.520782 11047 raft_consensus.cc:348] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.520880 11047 raft_consensus.cc:374] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.520916 11047 raft_consensus.cc:729] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.521034 11047 consensus_queue.cc:260] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.521302 11047 ts_tablet_manager.cc:1398] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:43.522790 11046 tablet_bootstrap.cc:492] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:43.522876 11046 ts_tablet_manager.cc:1367] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:43.523350 11046 raft_consensus.cc:348] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.523459 11046 raft_consensus.cc:374] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.523504 11046 raft_consensus.cc:729] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.523627 11046 consensus_queue.cc:260] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.523885 11046 ts_tablet_manager.cc:1398] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:43.566067 10891 raft_consensus.cc:482] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.566184 10891 raft_consensus.cc:504] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.566651 10891 leader_election.cc:288] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:43.567108 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1952714471f4771b28b78fe3b9a8c6b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:43.567256 9271 raft_consensus.cc:2454] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.567914 9470 leader_election.cc:302] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.568065 10891 raft_consensus.cc:2781] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.568120 10891 raft_consensus.cc:482] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.568157 10891 raft_consensus.cc:3037] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.569226 10891 raft_consensus.cc:504] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.569578 10891 leader_election.cc:288] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:43.569928 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1952714471f4771b28b78fe3b9a8c6b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:43.570032 9271 raft_consensus.cc:3037] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.571025 9271 raft_consensus.cc:2454] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:43.572788 9470 leader_election.cc:302] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.572959 10891 raft_consensus.cc:2781] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.573040 10891 raft_consensus.cc:686] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:43.573179 10891 consensus_queue.cc:237] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:43.575325 6737 catalog_manager.cc:5484] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:43.577670 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1952714471f4771b28b78fe3b9a8c6b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:43.577733 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1952714471f4771b28b78fe3b9a8c6b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:43.577803 7271 raft_consensus.cc:2454] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.591279 6988 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.592988 9251 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.602435 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:43.602500 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:43.603628 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:43.611585 9644 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:43.623126 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:43.623862 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.671306 6877 consensus_queue.cc:237] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.672480 7010 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.672610 9271 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.672824 10273 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.673002 11182 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.674633 11192 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.675019 7010 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.675138 9271 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.677647 6732 catalog_manager.cc:5484] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.680419 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4882718ff90c4d939917f4970a9acb9e with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:43.681022 11080 tablet_replica.cc:330] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:43.681169 11080 raft_consensus.cc:2229] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.681301 11080 raft_consensus.cc:2258] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.681782 11080 ts_tablet_manager.cc:1875] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.683630 11080 ts_tablet_manager.cc:1888] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:43.683709 11080 log.cc:1196] T 4882718ff90c4d939917f4970a9acb9e P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4882718ff90c4d939917f4970a9acb9e
I20240501 07:58:43.684366 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 4882718ff90c4d939917f4970a9acb9e (table rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]) successfully deleted
I20240501 07:58:43.687436 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:43.691867 7271 raft_consensus.cc:3037] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.691920 9271 raft_consensus.cc:1261] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.692422 10891 consensus_queue.cc:1035] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.692950 7271 raft_consensus.cc:1261] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:43.693320 10891 consensus_queue.cc:1035] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.695118 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]). Expected version 1 got 0
I20240501 07:58:43.696399 10968 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.696707 10968 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:43.697645 10967 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.697945 10967 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:43.698804 6730 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]). Expected version 1 got 0
I20240501 07:58:43.702062 6732 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce]). Expected version 1 got 0
I20240501 07:58:43.703253 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce] alter complete (version 1)
I20240501 07:58:43.707362 11208 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:43.708065 11208 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:43.732985 11207 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.736508 9735 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.746891 9734 consensus_queue.cc:1035] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.750141 9734 consensus_queue.cc:1035] T b4c8812331c847c9941e52daf0313897 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.774711 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 07:58:43.774856 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 07:58:43.778069 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 280d0dfa82ed4c91831a35e3e10f66d8
I20240501 07:58:43.778482 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 3a3a2b1e39454ba1a538e24e09d543ca
I20240501 07:58:43.778612 6988 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.778697 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet 48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:43.779562 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.779712 9251 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:43.780056 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.780261 11079 tablet_replica.cc:330] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.780390 11079 raft_consensus.cc:2229] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.780519 11079 raft_consensus.cc:2258] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.781103 11079 ts_tablet_manager.cc:1875] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.781464 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:43.781903 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:43.782959 11079 ts_tablet_manager.cc:1888] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.783085 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:43.783093 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:43.783250 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:43.783262 11082 tablet_replica.cc:330] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:43.783486 11082 raft_consensus.cc:2229] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.783605 11082 raft_consensus.cc:2258] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.783758 11078 tablet_replica.cc:330] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:43.783891 11078 raft_consensus.cc:2229] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.783982 11078 raft_consensus.cc:2258] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.784142 11082 ts_tablet_manager.cc:1875] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.784492 11078 ts_tablet_manager.cc:1875] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.786013 11082 ts_tablet_manager.cc:1888] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.786098 11082 log.cc:1196] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/280d0dfa82ed4c91831a35e3e10f66d8
I20240501 07:58:43.786298 11082 ts_tablet_manager.cc:1909] T 280d0dfa82ed4c91831a35e3e10f66d8 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:43.786505 11078 ts_tablet_manager.cc:1888] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.786573 11078 log.cc:1196] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:43.786823 11078 ts_tablet_manager.cc:1909] T 48964a6c57584c3eacc6dffac29d77a8 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:43.787349 11079 log.cc:1196] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a3a2b1e39454ba1a538e24e09d543ca
I20240501 07:58:43.787653 11079 ts_tablet_manager.cc:1909] T 3a3a2b1e39454ba1a538e24e09d543ca P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.787817 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:43.788007 11080 tablet_replica.cc:330] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:43.788079 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 280d0dfa82ed4c91831a35e3e10f66d8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.788138 11080 raft_consensus.cc:2229] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.788244 11079 tablet_replica.cc:330] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.788249 11082 tablet_replica.cc:330] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:43.788254 11080 raft_consensus.cc:2258] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.788349 11079 raft_consensus.cc:2229] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.788350 11082 raft_consensus.cc:2229] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.788481 11082 raft_consensus.cc:2258] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.788556 11079 raft_consensus.cc:2258] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.788736 11080 ts_tablet_manager.cc:1875] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.788933 11082 ts_tablet_manager.cc:1875] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.788993 11079 ts_tablet_manager.cc:1875] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.790784 11080 ts_tablet_manager.cc:1888] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.790848 11080 log.cc:1196] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3a3a2b1e39454ba1a538e24e09d543ca
I20240501 07:58:43.790923 11082 ts_tablet_manager.cc:1888] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.790941 11079 ts_tablet_manager.cc:1888] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.790994 11082 log.cc:1196] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:43.791285 11082 ts_tablet_manager.cc:1909] T 48964a6c57584c3eacc6dffac29d77a8 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:43.791363 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 3a3a2b1e39454ba1a538e24e09d543ca (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.791880 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:43.791976 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 48964a6c57584c3eacc6dffac29d77a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:43.792151 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3a2b1e39454ba1a538e24e09d543ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:43.780066 6988 tablet_service.cc:1514] Processing DeleteTablet for tablet 280d0dfa82ed4c91831a35e3e10f66d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:43 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.790994 11079 log.cc:1196] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:43.793033 11079 ts_tablet_manager.cc:1909] T 48964a6c57584c3eacc6dffac29d77a8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.793562 11079 tablet_replica.cc:330] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.793658 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 48964a6c57584c3eacc6dffac29d77a8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.793906 11080 ts_tablet_manager.cc:1909] T 3a3a2b1e39454ba1a538e24e09d543ca P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:43.794091 11079 raft_consensus.cc:2229] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.794257 11079 raft_consensus.cc:2258] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.794713 11079 ts_tablet_manager.cc:1875] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.794981 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 48964a6c57584c3eacc6dffac29d77a8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.795413 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 3a3a2b1e39454ba1a538e24e09d543ca (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.798171 11079 ts_tablet_manager.cc:1888] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.798535 11079 log.cc:1196] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/280d0dfa82ed4c91831a35e3e10f66d8
I20240501 07:58:43.798902 11079 ts_tablet_manager.cc:1909] T 280d0dfa82ed4c91831a35e3e10f66d8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:43.799177 11254 tablet_replica.cc:330] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:43.799348 11254 raft_consensus.cc:2229] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.799512 11254 raft_consensus.cc:2258] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.799676 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 280d0dfa82ed4c91831a35e3e10f66d8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.800030 11254 ts_tablet_manager.cc:1875] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.802088 11254 ts_tablet_manager.cc:1888] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.802165 11254 log.cc:1196] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/280d0dfa82ed4c91831a35e3e10f66d8
I20240501 07:58:43.802404 11254 ts_tablet_manager.cc:1909] T 280d0dfa82ed4c91831a35e3e10f66d8 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:43.802938 11254 tablet_replica.cc:330] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:43.803047 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 280d0dfa82ed4c91831a35e3e10f66d8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.803085 11254 raft_consensus.cc:2229] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.803223 11254 raft_consensus.cc:2258] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.803752 11254 ts_tablet_manager.cc:1875] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.804832 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:43.805724 11254 ts_tablet_manager.cc:1888] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:43.805810 11254 log.cc:1196] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/48964a6c57584c3eacc6dffac29d77a8
I20240501 07:58:43.806097 11254 ts_tablet_manager.cc:1909] T 48964a6c57584c3eacc6dffac29d77a8 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:43.806645 11254 tablet_replica.cc:330] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:43.806798 11254 raft_consensus.cc:2229] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:43.806996 11254 raft_consensus.cc:2258] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.807363 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 48964a6c57584c3eacc6dffac29d77a8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.807484 11254 ts_tablet_manager.cc:1875] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:43.807626 9644 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:43.809448 11254 ts_tablet_manager.cc:1888] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:43.809587 11254 log.cc:1196] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3a3a2b1e39454ba1a538e24e09d543ca
I20240501 07:58:43.810050 11254 ts_tablet_manager.cc:1909] T 3a3a2b1e39454ba1a538e24e09d543ca P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:43.811337 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 3a3a2b1e39454ba1a538e24e09d543ca (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.811993 10968 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.812280 10968 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:43.815683 11208 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.816037 11208 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:43.816721 10967 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:43.816988 10967 tablet.cc:1620] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:43.819903 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ce54546128324235a71e11ea031785ce] alter complete (version 2)
I20240501 07:58:43.831261 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:43.835965 10041 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.837172 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 48964a6c57584c3eacc6dffac29d77a8 (table rebalance_test_table_14 [id=c70f634763df42c8a6cc730777f0fda3]) successfully deleted
I20240501 07:58:43.841189 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:43.848595 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:43.849604 10041 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:43.870769 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:43.877209 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e1de6fc394fcb98c64165d74ca317 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.877456 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 0a81efefaf414a6091da247b0986c322 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.877533 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e1de6fc394fcb98c64165d74ca317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.877714 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a81efefaf414a6091da247b0986c322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.878134 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e1de6fc394fcb98c64165d74ca317 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.878451 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e1de6fc394fcb98c64165d74ca317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.879807 11045 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:43.879992 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 64e235bbc64a4dca8028ebf89c2b7b5e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.880187 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e235bbc64a4dca8028ebf89c2b7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.880757 11045 tablet_bootstrap.cc:654] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.881019 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e1de6fc394fcb98c64165d74ca317 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.881337 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e1de6fc394fcb98c64165d74ca317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.881772 11045 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:43.881850 11045 ts_tablet_manager.cc:1367] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:43.882263 11045 raft_consensus.cc:348] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.882287 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 0a81efefaf414a6091da247b0986c322 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.882369 11045 raft_consensus.cc:374] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.882416 11045 raft_consensus.cc:729] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.882529 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a81efefaf414a6091da247b0986c322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.882517 11045 consensus_queue.cc:260] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.882802 11045 ts_tablet_manager.cc:1398] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.882933 11045 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:43.882978 11047 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.883776 11045 tablet_bootstrap.cc:654] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.883875 11047 tablet_bootstrap.cc:654] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.884686 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 64e235bbc64a4dca8028ebf89c2b7b5e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.884898 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e235bbc64a4dca8028ebf89c2b7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.885010 6988 tablet_service.cc:1467] Processing CreateTablet for tablet 0a81efefaf414a6091da247b0986c322 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.885262 6988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a81efefaf414a6091da247b0986c322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.885690 11047 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.885697 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 64e235bbc64a4dca8028ebf89c2b7b5e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:43.885767 11047 ts_tablet_manager.cc:1367] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:43.885921 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e235bbc64a4dca8028ebf89c2b7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:43.886198 11047 raft_consensus.cc:348] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.886289 11047 raft_consensus.cc:374] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.886317 11047 raft_consensus.cc:729] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.886433 11047 consensus_queue.cc:260] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.886698 11047 ts_tablet_manager.cc:1398] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:43.886915 11047 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.887959 11047 tablet_bootstrap.cc:654] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.888974 11257 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:43.889770 11257 tablet_bootstrap.cc:654] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.891016 11257 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:43.891098 11257 ts_tablet_manager.cc:1367] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:43.891283 11047 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.891348 11047 ts_tablet_manager.cc:1367] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:43.891548 11257 raft_consensus.cc:348] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.891641 11257 raft_consensus.cc:374] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.891672 11257 raft_consensus.cc:729] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.891764 11047 raft_consensus.cc:348] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.891795 11257 consensus_queue.cc:260] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.891846 11047 raft_consensus.cc:374] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.891876 11047 raft_consensus.cc:729] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.892066 11047 consensus_queue.cc:260] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.892279 11257 ts_tablet_manager.cc:1398] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.892494 11257 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:43.892531 11047 ts_tablet_manager.cc:1398] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:43.892742 11047 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:43.893399 11257 tablet_bootstrap.cc:654] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.893638 11047 tablet_bootstrap.cc:654] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.895334 11047 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:43.895432 11047 ts_tablet_manager.cc:1367] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:43.895828 11047 raft_consensus.cc:348] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.895923 11047 raft_consensus.cc:374] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.895958 11047 raft_consensus.cc:729] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.896116 11047 consensus_queue.cc:260] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.896418 11047 ts_tablet_manager.cc:1398] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.899614 11264 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:43.900156 11045 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:43.900257 11045 ts_tablet_manager.cc:1367] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:58:43.900538 11264 tablet_bootstrap.cc:654] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.900743 11045 raft_consensus.cc:348] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.900887 11045 raft_consensus.cc:374] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.900954 11045 raft_consensus.cc:729] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.901120 11045 consensus_queue.cc:260] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.901461 11045 ts_tablet_manager.cc:1398] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:43.901548 11264 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:43.901641 11264 ts_tablet_manager.cc:1367] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.901643 11045 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:43.902542 11045 tablet_bootstrap.cc:654] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:43.903162 11264 raft_consensus.cc:348] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.903342 11264 raft_consensus.cc:374] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.903420 11264 raft_consensus.cc:729] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.903571 11264 consensus_queue.cc:260] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.904045 11264 ts_tablet_manager.cc:1398] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:43.904870 11257 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:43.904968 11257 ts_tablet_manager.cc:1367] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:43.905228 11045 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:43.905308 11045 ts_tablet_manager.cc:1367] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:43.905443 11257 raft_consensus.cc:348] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.905555 11257 raft_consensus.cc:374] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.905579 11257 raft_consensus.cc:729] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.905684 11257 consensus_queue.cc:260] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.905752 11045 raft_consensus.cc:348] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.905858 11045 raft_consensus.cc:374] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:43.905889 11045 raft_consensus.cc:729] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:43.905894 11257 ts_tablet_manager.cc:1398] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:43.906037 11045 consensus_queue.cc:260] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.906358 11045 ts_tablet_manager.cc:1398] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:43.920740 7140 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.921994 9271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.922381 7271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:43.922430 10346 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.922732 10777 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:43.924242 10346 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.924649 9271 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.925088 7271 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.926947 9735 raft_consensus.cc:482] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.927076 9735 raft_consensus.cc:504] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.927532 9735 leader_election.cc:288] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:43.927875 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:43.928150 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:43.928257 9271 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.928403 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:43.928521 7010 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:43.928543 9470 leader_election.cc:302] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.928721 6737 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.930586 9735 raft_consensus.cc:2781] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.930652 9735 raft_consensus.cc:482] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.930696 9735 raft_consensus.cc:3037] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.931663 9735 raft_consensus.cc:504] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.931980 9735 leader_election.cc:288] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:43.932888 6988 tablet_service.cc:1514] Processing DeleteTablet for tablet 80152bb09e4e42a09e106cf8b58178e3 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:43.933089 11079 tablet_replica.cc:330] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:43.933264 11079 raft_consensus.cc:2229] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.933281 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:43.933388 7010 raft_consensus.cc:3037] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.933461 11079 raft_consensus.cc:2258] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.933869 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:43.933962 9271 raft_consensus.cc:3037] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.934037 11079 ts_tablet_manager.cc:1875] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.934396 7010 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:43.935076 9471 leader_election.cc:302] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:43.935047 10968 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.935220 10891 raft_consensus.cc:2781] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.935281 10891 raft_consensus.cc:686] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:43.935333 10967 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.935405 10891 consensus_queue.cc:237] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:43.936048 10747 raft_consensus.cc:482] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:43.936128 11079 ts_tablet_manager.cc:1888] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:43.936127 10747 raft_consensus.cc:504] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.936188 11079 log.cc:1196] T 80152bb09e4e42a09e106cf8b58178e3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/80152bb09e4e42a09e106cf8b58178e3
I20240501 07:58:43.936460 10747 leader_election.cc:288] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.936733 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 80152bb09e4e42a09e106cf8b58178e3 (table rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]) successfully deleted
I20240501 07:58:43.938112 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e1de6fc394fcb98c64165d74ca317" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:43.938149 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e1de6fc394fcb98c64165d74ca317" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:43.938274 9535 raft_consensus.cc:2454] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:43.938271 9665 raft_consensus.cc:2454] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:43.938565 9206 leader_election.cc:302] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:43.938560 6730 catalog_manager.cc:5484] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:43.938707 10747 raft_consensus.cc:2781] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:43.938750 10747 raft_consensus.cc:482] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:43.938786 10747 raft_consensus.cc:3037] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.939762 11208 tablet.cc:1722] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:43.939869 10747 raft_consensus.cc:504] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.940227 10747 leader_election.cc:288] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:43.940672 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e1de6fc394fcb98c64165d74ca317" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:43.940773 9665 raft_consensus.cc:3037] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.941462 9665 raft_consensus.cc:2454] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:43.941728 9206 leader_election.cc:302] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:43.941915 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e1de6fc394fcb98c64165d74ca317" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:43.942009 9535 raft_consensus.cc:3037] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:43.942203 10747 raft_consensus.cc:2781] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:43.942291 10747 raft_consensus.cc:686] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:43.942414 10747 consensus_queue.cc:237] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:43.942798 9535 raft_consensus.cc:2454] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:43.944659 9271 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:43.944808 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=ce54546128324235a71e11ea031785ce] alter complete (version 3)
I20240501 07:58:43.946298 6737 catalog_manager.cc:5484] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.047222 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:44.047385 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:44.049453 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet a1952714471f4771b28b78fe3b9a8c6b
I20240501 07:58:44.050002 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet a1952714471f4771b28b78fe3b9a8c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:44.050040 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet a1952714471f4771b28b78fe3b9a8c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:44.050138 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet a1952714471f4771b28b78fe3b9a8c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:44.050215 11254 tablet_replica.cc:330] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:44.050381 11254 raft_consensus.cc:2229] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.050463 11330 tablet_replica.cc:330] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:44.050503 11254 raft_consensus.cc:2258] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.050614 11330 raft_consensus.cc:2229] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.050732 11078 tablet_replica.cc:330] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:44.050837 11330 raft_consensus.cc:2258] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.050879 11078 raft_consensus.cc:2229] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.051141 11078 raft_consensus.cc:2258] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.051137 11254 ts_tablet_manager.cc:1875] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.051489 11330 ts_tablet_manager.cc:1875] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.051715 11078 ts_tablet_manager.cc:1875] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.053592 11254 ts_tablet_manager.cc:1888] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.053674 11254 log.cc:1196] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1952714471f4771b28b78fe3b9a8c6b
I20240501 07:58:44.053716 11330 ts_tablet_manager.cc:1888] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.053769 11330 log.cc:1196] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a1952714471f4771b28b78fe3b9a8c6b
I20240501 07:58:44.053895 11078 ts_tablet_manager.cc:1888] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.053938 11254 ts_tablet_manager.cc:1909] T a1952714471f4771b28b78fe3b9a8c6b P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:44.053956 11078 log.cc:1196] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a1952714471f4771b28b78fe3b9a8c6b
I20240501 07:58:44.053969 11330 ts_tablet_manager.cc:1909] T a1952714471f4771b28b78fe3b9a8c6b P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:44.054190 11078 ts_tablet_manager.cc:1909] T a1952714471f4771b28b78fe3b9a8c6b P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:44.054689 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table_new_name [id=ce54546128324235a71e11ea031785ce]) successfully deleted
I20240501 07:58:44.054947 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table_new_name [id=ce54546128324235a71e11ea031785ce]) successfully deleted
I20240501 07:58:44.054992 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet a1952714471f4771b28b78fe3b9a8c6b (table rebalancer_dynamic_table_new_name [id=ce54546128324235a71e11ea031785ce]) successfully deleted
I20240501 07:58:44.150899 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:44.155956 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]), partition=
I20240501 07:58:44.156087 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]), partition=
I20240501 07:58:44.156230 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88daa98af15f47a6a60efcf4aa072a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.156314 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88daa98af15f47a6a60efcf4aa072a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.158390 11331 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:44.158560 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]), partition=
I20240501 07:58:44.158615 11045 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:44.158800 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88daa98af15f47a6a60efcf4aa072a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.159385 11331 tablet_bootstrap.cc:654] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.159444 11045 tablet_bootstrap.cc:654] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.160900 11045 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:44.160955 11257 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:44.160979 11045 ts_tablet_manager.cc:1367] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:44.161423 11045 raft_consensus.cc:348] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.161545 11045 raft_consensus.cc:374] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.161579 11045 raft_consensus.cc:729] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.161700 11045 consensus_queue.cc:260] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.161756 11257 tablet_bootstrap.cc:654] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.161960 11331 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:44.161960 11045 ts_tablet_manager.cc:1398] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.162034 11331 ts_tablet_manager.cc:1367] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:44.162904 11331 raft_consensus.cc:348] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.163060 11331 raft_consensus.cc:374] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.163120 11331 raft_consensus.cc:729] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.163266 11331 consensus_queue.cc:260] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.163612 11331 ts_tablet_manager.cc:1398] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:44.164634 10636 raft_consensus.cc:482] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:44.164711 10636 raft_consensus.cc:504] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:44.165067 10636 leader_election.cc:288] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:44.165390 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e235bbc64a4dca8028ebf89c2b7b5e" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:44.165416 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e235bbc64a4dca8028ebf89c2b7b5e" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:44.165505 9271 raft_consensus.cc:2454] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:44.165511 9535 raft_consensus.cc:2454] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:44.165788 9602 leader_election.cc:302] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:44.166018 10636 raft_consensus.cc:2781] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:44.166074 10636 raft_consensus.cc:482] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:44.166121 10636 raft_consensus.cc:3037] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.166884 10636 raft_consensus.cc:504] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:44.167860 10636 leader_election.cc:288] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:44.168141 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e235bbc64a4dca8028ebf89c2b7b5e" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:44.168255 9271 raft_consensus.cc:3037] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.168226 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e235bbc64a4dca8028ebf89c2b7b5e" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:44.168332 9535 raft_consensus.cc:3037] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.169184 9535 raft_consensus.cc:2454] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:44.169234 9271 raft_consensus.cc:2454] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:44.169508 9601 leader_election.cc:302] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:44.169548 11257 tablet_bootstrap.cc:492] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:44.169662 11257 ts_tablet_manager.cc:1367] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:44.169687 10636 raft_consensus.cc:2781] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:44.169764 10636 raft_consensus.cc:686] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:44.169868 10636 consensus_queue.cc:237] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:44.170117 11257 raft_consensus.cc:348] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.170210 11257 raft_consensus.cc:374] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.170245 11257 raft_consensus.cc:729] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.170379 11257 consensus_queue.cc:260] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.170668 11257 ts_tablet_manager.cc:1398] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.171542 6732 catalog_manager.cc:5484] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:44.194880 10891 raft_consensus.cc:482] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:44.196017 10891 raft_consensus.cc:504] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.196456 10891 leader_election.cc:288] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:44.196789 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88daa98af15f47a6a60efcf4aa072a4e" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:44.196933 9665 raft_consensus.cc:2454] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:44.196913 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88daa98af15f47a6a60efcf4aa072a4e" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:44.197062 9402 raft_consensus.cc:2454] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:44.197381 9470 leader_election.cc:302] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:44.198662 10891 raft_consensus.cc:2781] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:44.198719 10891 raft_consensus.cc:482] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:44.198753 10891 raft_consensus.cc:3037] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.199163 7250 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.199904 10891 raft_consensus.cc:504] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.200248 10891 leader_election.cc:288] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:44.202348 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88daa98af15f47a6a60efcf4aa072a4e" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:44.202464 9665 raft_consensus.cc:3037] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.202818 6988 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.203465 9665 raft_consensus.cc:2454] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:44.205735 9470 leader_election.cc:302] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:44.205899 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88daa98af15f47a6a60efcf4aa072a4e" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:44.205955 10891 raft_consensus.cc:2781] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:44.206022 10891 raft_consensus.cc:686] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:44.206023 9402 raft_consensus.cc:3037] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.206153 10891 consensus_queue.cc:237] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.206991 9402 raft_consensus.cc:2454] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:44.209453 6732 catalog_manager.cc:5484] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.213266 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.215091 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.216775 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.217393 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.234525 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:44.237229 9507 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:44.286180 7010 consensus_queue.cc:237] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.287312 9665 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.287393 6877 raft_consensus.cc:1261] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.287662 9936 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.287963 9936 consensus_queue.cc:1035] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.288606 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 07:58:44.288744 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 07:58:44.289605 9936 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.290131 9665 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.291636 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 4 replicas of tablet e05498632f0945c29f723a4613a26a04
I20240501 07:58:44.292025 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:44.292191 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet e05498632f0945c29f723a4613a26a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:44.292215 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet daa83d6382e340d28e60df08fd470d52
I20240501 07:58:44.292342 11080 tablet_replica.cc:330] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:44.292467 11080 raft_consensus.cc:2229] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.292678 11080 raft_consensus.cc:2258] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.292901 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet e05498632f0945c29f723a4613a26a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:44.292966 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3d410c640243848a478aa2605c1bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:44.293043 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3d410c640243848a478aa2605c1bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:44.293176 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet e05498632f0945c29f723a4613a26a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:44.293218 11080 ts_tablet_manager.cc:1875] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.293252 11366 tablet_replica.cc:330] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:44.293442 11366 raft_consensus.cc:2229] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.293521 11082 tablet_replica.cc:330] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:44.293550 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet daa83d6382e340d28e60df08fd470d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:44.293582 11366 raft_consensus.cc:2258] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.293653 11082 raft_consensus.cc:2229] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.293783 11082 raft_consensus.cc:2258] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.294013 11366 ts_tablet_manager.cc:1875] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.294289 11082 ts_tablet_manager.cc:1875] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.295822 11254 tablet_replica.cc:330] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:44.295989 11254 raft_consensus.cc:2229] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.296115 11254 raft_consensus.cc:2258] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.296401 11082 ts_tablet_manager.cc:1888] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.296378 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet daa83d6382e340d28e60df08fd470d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:44.296465 11082 log.cc:1196] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e05498632f0945c29f723a4613a26a04
I20240501 07:58:44.296680 11254 ts_tablet_manager.cc:1875] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.296746 11082 ts_tablet_manager.cc:1909] T e05498632f0945c29f723a4613a26a04 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:44.296897 11366 ts_tablet_manager.cc:1888] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:58:44.296963 11366 log.cc:1196] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:44.297286 11082 tablet_replica.cc:330] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:44.297286 11366 ts_tablet_manager.cc:1909] T 0d3d410c640243848a478aa2605c1bf0 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:44.297385 11082 raft_consensus.cc:2229] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.297497 11082 raft_consensus.cc:2258] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.297509 11080 ts_tablet_manager.cc:1888] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.297571 11080 log.cc:1196] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e05498632f0945c29f723a4613a26a04
I20240501 07:58:44.297834 11080 ts_tablet_manager.cc:1909] T e05498632f0945c29f723a4613a26a04 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:44.297940 11082 ts_tablet_manager.cc:1875] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.298404 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet e05498632f0945c29f723a4613a26a04 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.298324 11080 tablet_replica.cc:330] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:44.298856 11254 ts_tablet_manager.cc:1888] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.298895 11080 raft_consensus.cc:2229] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.298938 11254 log.cc:1196] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/daa83d6382e340d28e60df08fd470d52
I20240501 07:58:44.299090 11080 raft_consensus.cc:2258] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.299111 11078 tablet_replica.cc:330] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:44.299237 11078 raft_consensus.cc:2229] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.299337 11078 raft_consensus.cc:2258] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.299212 6735 catalog_manager.cc:5484] T 58aab13671d647e485f9be94283b5239 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.299539 11080 ts_tablet_manager.cc:1875] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.299834 11078 ts_tablet_manager.cc:1875] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.301376 11080 ts_tablet_manager.cc:1888] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.301633 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 58aab13671d647e485f9be94283b5239 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:44.301440 11080 log.cc:1196] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/daa83d6382e340d28e60df08fd470d52
I20240501 07:58:44.301928 11078 ts_tablet_manager.cc:1888] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.301950 11080 ts_tablet_manager.cc:1909] T daa83d6382e340d28e60df08fd470d52 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:44.301991 11078 log.cc:1196] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e05498632f0945c29f723a4613a26a04
I20240501 07:58:44.302240 11078 ts_tablet_manager.cc:1909] T e05498632f0945c29f723a4613a26a04 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:44.302410 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet daa83d6382e340d28e60df08fd470d52 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.302539 11080 tablet_replica.cc:330] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:44.302565 11082 ts_tablet_manager.cc:1888] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:58:44.302624 11082 log.cc:1196] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:44.302623 11080 raft_consensus.cc:2229] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.302711 11080 raft_consensus.cc:2258] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.302889 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet e05498632f0945c29f723a4613a26a04 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.303155 11082 ts_tablet_manager.cc:1909] T 0d3d410c640243848a478aa2605c1bf0 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:44.303131 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet e05498632f0945c29f723a4613a26a04 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.303182 11080 ts_tablet_manager.cc:1875] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.303242 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 0d3d410c640243848a478aa2605c1bf0 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.304117 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 0d3d410c640243848a478aa2605c1bf0 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.304253 6988 tablet_service.cc:1514] Processing DeleteTablet for tablet e05498632f0945c29f723a4613a26a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.304394 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3d410c640243848a478aa2605c1bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.304539 11079 tablet_replica.cc:330] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.304476 6877 raft_consensus.cc:2932] T 58aab13671d647e485f9be94283b5239 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.304684 11079 raft_consensus.cc:2229] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.304769 11254 ts_tablet_manager.cc:1909] T daa83d6382e340d28e60df08fd470d52 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:44.304819 11079 raft_consensus.cc:2258] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.305285 11079 ts_tablet_manager.cc:1875] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.305418 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet daa83d6382e340d28e60df08fd470d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.305507 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet daa83d6382e340d28e60df08fd470d52 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.306974 11079 ts_tablet_manager.cc:1888] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.307055 11079 log.cc:1196] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e05498632f0945c29f723a4613a26a04
I20240501 07:58:44.307312 11079 ts_tablet_manager.cc:1909] T e05498632f0945c29f723a4613a26a04 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.307818 11079 tablet_replica.cc:330] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.307937 11079 raft_consensus.cc:2229] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.308087 11079 raft_consensus.cc:2258] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.308493 11079 ts_tablet_manager.cc:1875] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.308533 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet e05498632f0945c29f723a4613a26a04 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.309160 11080 ts_tablet_manager.cc:1888] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:44.309237 11080 log.cc:1196] T 58aab13671d647e485f9be94283b5239 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/58aab13671d647e485f9be94283b5239
I20240501 07:58:44.309795 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 58aab13671d647e485f9be94283b5239 (table rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]) successfully deleted
I20240501 07:58:44.310110 11079 ts_tablet_manager.cc:1888] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:58:44.310169 11079 log.cc:1196] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d3d410c640243848a478aa2605c1bf0
I20240501 07:58:44.310389 11079 ts_tablet_manager.cc:1909] T 0d3d410c640243848a478aa2605c1bf0 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.310914 11079 tablet_replica.cc:330] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.311023 11079 raft_consensus.cc:2229] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.311030 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 0d3d410c640243848a478aa2605c1bf0 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.311134 11079 raft_consensus.cc:2258] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.311563 11079 ts_tablet_manager.cc:1875] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.312990 11079 ts_tablet_manager.cc:1888] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.313052 11079 log.cc:1196] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/daa83d6382e340d28e60df08fd470d52
I20240501 07:58:44.313292 11079 ts_tablet_manager.cc:1909] T daa83d6382e340d28e60df08fd470d52 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.313863 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet daa83d6382e340d28e60df08fd470d52 (table rebalance_test_table_16 [id=27ab3d3d5f18475fa304dba5270c4187]) successfully deleted
I20240501 07:58:44.319478 9735 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.322454 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:44.322471 9735 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.335497 9665 raft_consensus.cc:1261] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:44.335862 9402 raft_consensus.cc:1261] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:44.336261 9735 consensus_queue.cc:1035] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.336392 10891 consensus_queue.cc:1035] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.339367 6735 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]). Expected version 1 got 0
I20240501 07:58:44.341331 10968 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:44.341642 10968 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:44.340209 6732 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]). Expected version 1 got 0
I20240501 07:58:44.344359 6735 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]). Expected version 1 got 0
I20240501 07:58:44.347246 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418] alter complete (version 1)
I20240501 07:58:44.350654 6735 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418]). Expected version 1 got 0
I20240501 07:58:44.351296 11383 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:44.351868 11383 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:44.352051 11385 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:44.352442 11385 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:44.405997 10747 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.413275 10747 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.417193 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.417407 7250 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.419356 9507 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:44.425293 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.433231 6990 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.438305 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.448371 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.449781 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.451218 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:44.465561 10968 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:44.465912 10968 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:44.469653 11383 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:44.469959 11383 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:44.472680 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=4636503eeb564f7994ee3fa9c1871418] alter complete (version 2)
I20240501 07:58:44.473251 11385 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:44.473574 11385 tablet.cc:1620] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:44.508999 7140 consensus_queue.cc:237] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:44.510218 9535 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.510244 7271 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.510643 10346 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.510793 10777 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.512599 10346 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:44.513028 7271 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:44.513512 9535 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:44.516013 6737 catalog_manager.cc:5484] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:44.519402 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet c80723cb0ce4446483a5f1754e066081 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.519734 11079 tablet_replica.cc:330] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.519870 11079 raft_consensus.cc:2229] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.520028 11079 raft_consensus.cc:2258] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.520561 11079 ts_tablet_manager.cc:1875] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.522704 11079 ts_tablet_manager.cc:1888] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:44.522773 11079 log.cc:1196] T c80723cb0ce4446483a5f1754e066081 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c80723cb0ce4446483a5f1754e066081
I20240501 07:58:44.524312 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet c80723cb0ce4446483a5f1754e066081 (table rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]) successfully deleted
I20240501 07:58:44.570078 9877 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.574352 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:44.574846 10636 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:44.580245 10968 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:44.582808 11383 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:44.586505 11385 tablet.cc:1722] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:44.588486 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=4636503eeb564f7994ee3fa9c1871418] alter complete (version 3)
I20240501 07:58:44.615648 7250 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.617038 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.618258 6990 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.620348 9507 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:44.622756 6858 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.635170 9381 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.637480 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.646492 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.686996 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:44.693272 9250 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0b9c5ac80488897e07511c3006c7f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.693619 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0b9c5ac80488897e07511c3006c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.693993 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.694255 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.694265 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 0c86818cabaf49678cfbcce3480f2645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.694556 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c86818cabaf49678cfbcce3480f2645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.696285 9507 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0b9c5ac80488897e07511c3006c7f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.696583 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0b9c5ac80488897e07511c3006c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.696871 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.697127 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.698884 11515 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:44.699177 6858 tablet_service.cc:1467] Processing CreateTablet for tablet e9e0b9c5ac80488897e07511c3006c7f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.699262 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 0c86818cabaf49678cfbcce3480f2645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.699448 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e0b9c5ac80488897e07511c3006c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.699469 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c86818cabaf49678cfbcce3480f2645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.700264 11515 tablet_bootstrap.cc:654] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.701505 11517 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:44.701900 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.702196 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 145b8afede0a4cfb8d9ee3d4eb445c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.702448 11517 tablet_bootstrap.cc:654] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.703488 11517 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:44.703547 11514 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:44.703562 11517 ts_tablet_manager.cc:1367] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:44.703980 11517 raft_consensus.cc:348] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.704072 11517 raft_consensus.cc:374] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.704098 11517 raft_consensus.cc:729] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.704198 11517 consensus_queue.cc:260] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.704434 11517 ts_tablet_manager.cc:1398] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.704555 11514 tablet_bootstrap.cc:654] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.704583 11517 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:44.704756 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 0c86818cabaf49678cfbcce3480f2645 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:44.705001 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c86818cabaf49678cfbcce3480f2645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.705143 11518 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:44.705363 11517 tablet_bootstrap.cc:654] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.705901 11518 tablet_bootstrap.cc:654] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.706933 11517 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:44.707026 11517 ts_tablet_manager.cc:1367] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:44.707486 11517 raft_consensus.cc:348] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.707604 11517 raft_consensus.cc:374] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.707634 11517 raft_consensus.cc:729] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.707677 11518 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:44.707749 11518 ts_tablet_manager.cc:1367] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:44.707741 11517 consensus_queue.cc:260] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.707971 11517 ts_tablet_manager.cc:1398] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.708144 11518 raft_consensus.cc:348] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.708248 11518 raft_consensus.cc:374] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.708281 11518 raft_consensus.cc:729] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.708456 11518 consensus_queue.cc:260] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.708714 11518 ts_tablet_manager.cc:1398] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.708851 11518 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:44.709923 11518 tablet_bootstrap.cc:654] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.710769 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:44.710909 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:44.711696 11518 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:44.711783 11518 ts_tablet_manager.cc:1367] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:44.712373 11518 raft_consensus.cc:348] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.712739 11518 raft_consensus.cc:374] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.712783 11518 raft_consensus.cc:729] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.713001 11518 consensus_queue.cc:260] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.713376 11518 ts_tablet_manager.cc:1398] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:44.713403 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 88daa98af15f47a6a60efcf4aa072a4e
I20240501 07:58:44.714074 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:44.714231 11366 tablet_replica.cc:330] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:44.714358 11366 raft_consensus.cc:2229] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.714478 11366 raft_consensus.cc:2258] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.714493 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:44.714699 11078 tablet_replica.cc:330] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:44.714839 11078 raft_consensus.cc:2229] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.715094 11078 raft_consensus.cc:2258] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.715126 11366 ts_tablet_manager.cc:1875] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.715839 11078 ts_tablet_manager.cc:1875] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.716145 11513 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:44.713030 11520 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:44.714262 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 88daa98af15f47a6a60efcf4aa072a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:44.717157 11366 ts_tablet_manager.cc:1888] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.717206 11366 log.cc:1196] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/88daa98af15f47a6a60efcf4aa072a4e
I20240501 07:58:44.717435 11366 ts_tablet_manager.cc:1909] T 88daa98af15f47a6a60efcf4aa072a4e P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:44.717741 11520 tablet_bootstrap.cc:654] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.717949 11078 ts_tablet_manager.cc:1888] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.718020 11078 log.cc:1196] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/88daa98af15f47a6a60efcf4aa072a4e
I20240501 07:58:44.718163 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table_new_name [id=4636503eeb564f7994ee3fa9c1871418]) successfully deleted
I20240501 07:58:44.719426 11513 tablet_bootstrap.cc:654] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.719614 11078 ts_tablet_manager.cc:1909] T 88daa98af15f47a6a60efcf4aa072a4e P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:44.721009 11514 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:44.721108 11514 ts_tablet_manager.cc:1367] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:58:44.721607 11514 raft_consensus.cc:348] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.721717 11514 raft_consensus.cc:374] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.721753 11514 raft_consensus.cc:729] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.721879 11514 consensus_queue.cc:260] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.722059 11513 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:44.722191 11514 ts_tablet_manager.cc:1398] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.722318 11513 ts_tablet_manager.cc:1367] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:44.722762 11513 raft_consensus.cc:348] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.722885 11513 raft_consensus.cc:374] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.722918 11513 raft_consensus.cc:729] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.723186 11514 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:44.723621 11513 consensus_queue.cc:260] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.723990 11513 ts_tablet_manager.cc:1398] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:44.724074 11514 tablet_bootstrap.cc:654] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.725044 11515 tablet_bootstrap.cc:492] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:44.725145 11515 ts_tablet_manager.cc:1367] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:58:44.725405 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table_new_name [id=4636503eeb564f7994ee3fa9c1871418]) successfully deleted
I20240501 07:58:44.725665 11515 raft_consensus.cc:348] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.725776 11515 raft_consensus.cc:374] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.725807 11515 raft_consensus.cc:729] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.725929 11515 consensus_queue.cc:260] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.726190 11515 ts_tablet_manager.cc:1398] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.728425 11526 tablet_replica.cc:330] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:44.728641 11526 raft_consensus.cc:2229] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.728839 11526 raft_consensus.cc:2258] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.729547 11526 ts_tablet_manager.cc:1875] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.730554 6877 consensus_queue.cc:237] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.732024 9403 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.732115 11526 ts_tablet_manager.cc:1888] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:44.732183 11526 log.cc:1196] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/88daa98af15f47a6a60efcf4aa072a4e
I20240501 07:58:44.732733 11520 tablet_bootstrap.cc:492] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:44.732795 7010 raft_consensus.cc:1261] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.732872 11514 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:44.732874 11436 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.732914 11520 ts_tablet_manager.cc:1367] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240501 07:58:44.732955 11514 ts_tablet_manager.cc:1367] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:44.733201 11526 ts_tablet_manager.cc:1909] T 88daa98af15f47a6a60efcf4aa072a4e P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:44.733567 11514 raft_consensus.cc:348] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.733616 11520 raft_consensus.cc:348] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.733707 11514 raft_consensus.cc:374] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.733711 11520 raft_consensus.cc:374] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.733752 11520 raft_consensus.cc:729] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.733752 11514 raft_consensus.cc:729] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.733871 11520 consensus_queue.cc:260] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.733938 11514 consensus_queue.cc:260] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.734076 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 88daa98af15f47a6a60efcf4aa072a4e (table rebalancer_dynamic_table_new_name [id=4636503eeb564f7994ee3fa9c1871418]) successfully deleted
I20240501 07:58:44.734303 11436 consensus_queue.cc:1035] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.734362 11514 ts_tablet_manager.cc:1398] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.736351 11436 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:44.736886 11520 ts_tablet_manager.cc:1398] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:44.739207 7010 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:44.739321 9403 raft_consensus.cc:2932] T 43b82e035ecb4e23973a2163e15c1dac P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:44.739492 6736 catalog_manager.cc:5484] T 43b82e035ecb4e23973a2163e15c1dac P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:44.742111 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b82e035ecb4e23973a2163e15c1dac with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:44.742316 11080 tablet_replica.cc:330] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:44.742457 11080 raft_consensus.cc:2229] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.742623 11080 raft_consensus.cc:2258] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.745734 11080 ts_tablet_manager.cc:1875] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.748356 11080 ts_tablet_manager.cc:1888] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:44.748651 11080 log.cc:1196] T 43b82e035ecb4e23973a2163e15c1dac P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/43b82e035ecb4e23973a2163e15c1dac
I20240501 07:58:44.749145 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 43b82e035ecb4e23973a2163e15c1dac (table rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]) successfully deleted
I20240501 07:58:44.795348 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 07:58:44.795490 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 07:58:44.798684 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 35eb30b03de54bccb36f5b8093abf4e3
I20240501 07:58:44.799052 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet c1889e43466f46018b95673d30fdd118
I20240501 07:58:44.799304 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet f0c3cb8aafcd4ab7be9018e6c63f0e50
I20240501 07:58:44.799397 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.799585 11079 tablet_replica.cc:330] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.799731 11079 raft_consensus.cc:2229] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.799863 11079 raft_consensus.cc:2258] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.799818 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:44.800024 11082 tablet_replica.cc:330] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:44.800153 11082 raft_consensus.cc:2229] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.800364 11082 raft_consensus.cc:2258] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.800436 11079 ts_tablet_manager.cc:1875] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.800602 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.800937 11082 ts_tablet_manager.cc:1875] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.802343 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet c1889e43466f46018b95673d30fdd118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:44.802353 11079 ts_tablet_manager.cc:1888] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.802457 11079 log.cc:1196] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/35eb30b03de54bccb36f5b8093abf4e3
I20240501 07:58:44.802984 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet c1889e43466f46018b95673d30fdd118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:44.803193 11080 tablet_replica.cc:330] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:44.803328 11080 raft_consensus.cc:2229] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.803375 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eb30b03de54bccb36f5b8093abf4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:44.803511 11080 raft_consensus.cc:2258] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.803573 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet c1889e43466f46018b95673d30fdd118 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:44.803624 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:44.803715 11254 tablet_replica.cc:330] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:44.803762 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:44 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:44.803843 11254 raft_consensus.cc:2229] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.803978 11080 ts_tablet_manager.cc:1875] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.804000 11254 raft_consensus.cc:2258] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.804420 11082 ts_tablet_manager.cc:1888] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.804491 11082 log.cc:1196] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/35eb30b03de54bccb36f5b8093abf4e3
I20240501 07:58:44.804585 11254 ts_tablet_manager.cc:1875] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.804772 11082 ts_tablet_manager.cc:1909] T 35eb30b03de54bccb36f5b8093abf4e3 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:44.805408 11082 tablet_replica.cc:330] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:44.805541 11082 raft_consensus.cc:2229] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.805747 11082 raft_consensus.cc:2258] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.806053 11080 ts_tablet_manager.cc:1888] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.806102 11080 log.cc:1196] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c1889e43466f46018b95673d30fdd118
I20240501 07:58:44.806248 11082 ts_tablet_manager.cc:1875] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.806305 11080 ts_tablet_manager.cc:1909] T c1889e43466f46018b95673d30fdd118 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:44.806833 11254 ts_tablet_manager.cc:1888] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.806914 11254 log.cc:1196] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/35eb30b03de54bccb36f5b8093abf4e3
I20240501 07:58:44.806970 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 35eb30b03de54bccb36f5b8093abf4e3 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.802759 11079 ts_tablet_manager.cc:1909] T 35eb30b03de54bccb36f5b8093abf4e3 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.807804 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet c1889e43466f46018b95673d30fdd118 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.807972 11254 ts_tablet_manager.cc:1909] T 35eb30b03de54bccb36f5b8093abf4e3 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:44.807986 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 35eb30b03de54bccb36f5b8093abf4e3 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.808187 11079 tablet_replica.cc:330] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.808332 11079 raft_consensus.cc:2229] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.808471 11079 raft_consensus.cc:2258] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.808504 11060 raft_consensus.cc:482] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:44.808629 11060 raft_consensus.cc:504] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.808940 11079 ts_tablet_manager.cc:1875] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.809132 11060 leader_election.cc:288] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:44.809393 10706 raft_consensus.cc:482] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:44.809491 10706 raft_consensus.cc:504] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.809620 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145b8afede0a4cfb8d9ee3d4eb445c1d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:44.809731 7010 raft_consensus.cc:2454] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:44.809870 10706 leader_election.cc:288] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:44.809931 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 35eb30b03de54bccb36f5b8093abf4e3 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.810058 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145b8afede0a4cfb8d9ee3d4eb445c1d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:44.810204 9665 raft_consensus.cc:2454] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:44.810480 9339 leader_election.cc:302] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:44.810667 11060 raft_consensus.cc:2781] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:44.810724 11060 raft_consensus.cc:482] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:44.810765 11060 raft_consensus.cc:3037] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.810770 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c86818cabaf49678cfbcce3480f2645" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:44.810895 6877 raft_consensus.cc:2454] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:44.810995 11079 ts_tablet_manager.cc:1888] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.811084 11079 log.cc:1196] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0c3cb8aafcd4ab7be9018e6c63f0e50
I20240501 07:58:44.811089 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c86818cabaf49678cfbcce3480f2645" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:44.811199 9403 raft_consensus.cc:2454] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:44.811293 11079 ts_tablet_manager.cc:1909] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.811540 6945 leader_election.cc:302] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:44.811712 10706 raft_consensus.cc:2781] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:44.811761 10706 raft_consensus.cc:482] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:44.811796 10706 raft_consensus.cc:3037] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.811872 11079 tablet_replica.cc:330] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:44.811849 11060 raft_consensus.cc:504] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.811990 11079 raft_consensus.cc:2229] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:44.812134 11079 raft_consensus.cc:2258] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.812214 11060 leader_election.cc:288] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:44.812624 11079 ts_tablet_manager.cc:1875] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.812631 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145b8afede0a4cfb8d9ee3d4eb445c1d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:44.812695 10706 raft_consensus.cc:504] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.812747 9665 raft_consensus.cc:3037] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.812996 10706 leader_election.cc:288] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:44.813084 11082 ts_tablet_manager.cc:1888] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.813148 11082 log.cc:1196] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f0c3cb8aafcd4ab7be9018e6c63f0e50
I20240501 07:58:44.813308 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.813328 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "145b8afede0a4cfb8d9ee3d4eb445c1d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:44.813422 7010 raft_consensus.cc:3037] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.813789 9665 raft_consensus.cc:2454] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:44.814338 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c86818cabaf49678cfbcce3480f2645" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:44.814371 7010 raft_consensus.cc:2454] T 145b8afede0a4cfb8d9ee3d4eb445c1d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:44.814348 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c86818cabaf49678cfbcce3480f2645" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:44.814433 6877 raft_consensus.cc:3037] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.814461 9403 raft_consensus.cc:3037] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.814491 11079 ts_tablet_manager.cc:1888] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.814550 11079 log.cc:1196] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c1889e43466f46018b95673d30fdd118
I20240501 07:58:44.814783 11082 ts_tablet_manager.cc:1909] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:44.814801 11079 ts_tablet_manager.cc:1909] T c1889e43466f46018b95673d30fdd118 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:44.815440 9403 raft_consensus.cc:2454] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:44.815559 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.815668 9339 leader_election.cc:302] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:44.815725 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet c1889e43466f46018b95673d30fdd118 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.815843 11060 raft_consensus.cc:2781] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:44.815853 6945 leader_election.cc:302] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:44.816007 11060 raft_consensus.cc:686] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:44.816156 10706 raft_consensus.cc:2781] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:44.816159 11060 consensus_queue.cc:237] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.816243 10706 raft_consensus.cc:686] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:44.816272 6877 raft_consensus.cc:2454] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:44.816365 10706 consensus_queue.cc:237] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:44.816665 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:44.817550 11254 tablet_replica.cc:330] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:44.817708 11254 raft_consensus.cc:2229] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.818064 11254 raft_consensus.cc:2258] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.818554 6736 catalog_manager.cc:5484] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:44.818903 11254 ts_tablet_manager.cc:1875] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.818851 6731 catalog_manager.cc:5484] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:44.821064 11254 ts_tablet_manager.cc:1888] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.821195 11254 log.cc:1196] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c1889e43466f46018b95673d30fdd118
I20240501 07:58:44.821552 11254 ts_tablet_manager.cc:1909] T c1889e43466f46018b95673d30fdd118 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:44.822317 11254 tablet_replica.cc:330] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:44.822562 11254 raft_consensus.cc:2229] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.822764 11254 raft_consensus.cc:2258] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.826498 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 2828bc2999bb4a6a84a9213486981e85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]), partition=
I20240501 07:58:44.826617 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 2828bc2999bb4a6a84a9213486981e85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]), partition=
I20240501 07:58:44.826761 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2828bc2999bb4a6a84a9213486981e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.826805 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2828bc2999bb4a6a84a9213486981e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.828667 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 2828bc2999bb4a6a84a9213486981e85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]), partition=
I20240501 07:58:44.828912 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2828bc2999bb4a6a84a9213486981e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:44.829093 11514 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:44.829149 11254 ts_tablet_manager.cc:1875] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:44.829218 11513 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:44.830027 11513 tablet_bootstrap.cc:654] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.830027 11514 tablet_bootstrap.cc:654] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.831149 11517 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:44.831280 11254 ts_tablet_manager.cc:1888] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:58:44.831353 11254 log.cc:1196] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f0c3cb8aafcd4ab7be9018e6c63f0e50
I20240501 07:58:44.831646 11254 ts_tablet_manager.cc:1909] T f0c3cb8aafcd4ab7be9018e6c63f0e50 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:44.832325 11517 tablet_bootstrap.cc:654] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:44.834059 11514 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:44.834137 11514 ts_tablet_manager.cc:1367] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:44.834618 11514 raft_consensus.cc:348] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.834717 11514 raft_consensus.cc:374] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.834735 11517 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:44.834753 11514 raft_consensus.cc:729] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.834811 11517 ts_tablet_manager.cc:1367] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:44.834873 11514 consensus_queue.cc:260] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.835294 11517 raft_consensus.cc:348] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.835393 11517 raft_consensus.cc:374] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.835425 11517 raft_consensus.cc:729] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.835429 11514 ts_tablet_manager.cc:1398] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.835544 11517 consensus_queue.cc:260] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.835812 11517 ts_tablet_manager.cc:1398] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.835788 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet c1889e43466f46018b95673d30fdd118 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.835945 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet f0c3cb8aafcd4ab7be9018e6c63f0e50 (table rebalance_test_table_18 [id=4f30bc26da4c4582b723e1ab47586a8e]) successfully deleted
I20240501 07:58:44.838696 11513 tablet_bootstrap.cc:492] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:44.838783 11513 ts_tablet_manager.cc:1367] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:44.839319 11513 raft_consensus.cc:348] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.839418 11513 raft_consensus.cc:374] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:44.839449 11513 raft_consensus.cc:729] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:44.839576 11513 consensus_queue.cc:260] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:44.839872 11513 ts_tablet_manager.cc:1398] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:44.852886 11182 raft_consensus.cc:482] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:44.852998 11182 raft_consensus.cc:504] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.853499 11182 leader_election.cc:288] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:44.854586 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0b9c5ac80488897e07511c3006c7f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:44.854735 9271 raft_consensus.cc:2454] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:44.854997 6814 leader_election.cc:302] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:44.855202 11182 raft_consensus.cc:2781] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:44.855252 11182 raft_consensus.cc:482] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:44.855288 11182 raft_consensus.cc:3037] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.856396 11182 raft_consensus.cc:504] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.856775 11182 leader_election.cc:288] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:44.857087 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0b9c5ac80488897e07511c3006c7f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:44.857194 9271 raft_consensus.cc:3037] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:44.858119 9271 raft_consensus.cc:2454] T e9e0b9c5ac80488897e07511c3006c7f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:44.858449 6814 leader_election.cc:302] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:44.858536 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0b9c5ac80488897e07511c3006c7f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:44.858621 11182 raft_consensus.cc:2781] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:44.858592 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e0b9c5ac80488897e07511c3006c7f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:44.858667 9535 raft_consensus.cc:2454] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:44.858711 11182 raft_consensus.cc:686] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:44.858834 11182 consensus_queue.cc:237] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:44.860579 6736 catalog_manager.cc:5484] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:44.877225 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.877475 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.880039 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:44.888605 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:44.891206 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:44.892513 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.898005 9381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:44.908859 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:44.962448 7271 consensus_queue.cc:237] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:44.963686 7138 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.963722 9665 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.964093 11116 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.964298 10041 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.966082 10041 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.966647 9665 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.967794 7138 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.969933 6732 catalog_manager.cc:5484] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:44.973467 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7dcdd38eff43fe9fff9081ee644c8d with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:44.973682 11082 tablet_replica.cc:330] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:44.973836 11082 raft_consensus.cc:2229] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.973989 11082 raft_consensus.cc:2258] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.974515 11082 ts_tablet_manager.cc:1875] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.976771 11082 ts_tablet_manager.cc:1888] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:44.976887 11082 log.cc:1196] T ab7dcdd38eff43fe9fff9081ee644c8d P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ab7dcdd38eff43fe9fff9081ee644c8d
I20240501 07:58:44.977409 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet ab7dcdd38eff43fe9fff9081ee644c8d (table rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]) successfully deleted
I20240501 07:58:45.085011 9936 raft_consensus.cc:482] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:45.085146 9936 raft_consensus.cc:504] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:45.085669 9936 leader_election.cc:288] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:45.086364 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2828bc2999bb4a6a84a9213486981e85" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:45.086529 9403 raft_consensus.cc:2454] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:45.086820 6945 leader_election.cc:302] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:45.086992 9936 raft_consensus.cc:2781] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:45.087069 9936 raft_consensus.cc:482] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:45.087106 9936 raft_consensus.cc:3037] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.088053 9936 raft_consensus.cc:504] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:45.088333 9936 leader_election.cc:288] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:45.088646 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2828bc2999bb4a6a84a9213486981e85" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:45.088761 9403 raft_consensus.cc:3037] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.089272 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2828bc2999bb4a6a84a9213486981e85" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:45.089416 9271 raft_consensus.cc:2454] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:45.089311 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2828bc2999bb4a6a84a9213486981e85" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:45.089661 9266 raft_consensus.cc:3037] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.089666 9403 raft_consensus.cc:2454] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:45.090435 6945 leader_election.cc:302] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:45.090481 9266 raft_consensus.cc:2454] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:45.090646 9936 raft_consensus.cc:2781] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.090764 9936 raft_consensus.cc:686] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:45.090935 9936 consensus_queue.cc:237] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:45.092873 6730 catalog_manager.cc:5484] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.108246 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.114964 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.121956 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:45.122067 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.124591 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.127753 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.128027 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.128330 9381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:45.197145 6877 consensus_queue.cc:237] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:45.198714 7010 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:45.199172 9403 raft_consensus.cc:1261] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:45.199180 11436 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:45.199676 11436 consensus_queue.cc:1035] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:45.201157 11182 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.201843 7010 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.203449 9403 raft_consensus.cc:2932] T 2304f5e5d09a47b793cbb4f60b255525 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.205830 6730 catalog_manager.cc:5484] T 2304f5e5d09a47b793cbb4f60b255525 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.208948 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 2304f5e5d09a47b793cbb4f60b255525 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:45.209228 11080 tablet_replica.cc:330] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:45.209396 11080 raft_consensus.cc:2229] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:45.209545 11080 raft_consensus.cc:2258] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:45.210072 11080 ts_tablet_manager.cc:1875] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:45.212028 11080 ts_tablet_manager.cc:1888] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:45.212087 11080 log.cc:1196] T 2304f5e5d09a47b793cbb4f60b255525 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2304f5e5d09a47b793cbb4f60b255525
I20240501 07:58:45.212659 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 2304f5e5d09a47b793cbb4f60b255525 (table rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]) successfully deleted
I20240501 07:58:45.218972 11060 consensus_queue.cc:1035] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.224447 11060 consensus_queue.cc:1035] T 145b8afede0a4cfb8d9ee3d4eb445c1d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.242218 11700 consensus_queue.cc:1035] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.245347 9534 raft_consensus.cc:3037] T e9e0b9c5ac80488897e07511c3006c7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.246994 11700 consensus_queue.cc:1035] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.250223 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:45.254304 9403 raft_consensus.cc:1261] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.254419 9266 raft_consensus.cc:1261] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.254700 9936 consensus_queue.cc:1035] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.254837 10706 consensus_queue.cc:1035] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.257328 6732 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]). Expected version 1 got 0
I20240501 07:58:45.257977 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]). Expected version 1 got 0
I20240501 07:58:45.261112 6733 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]). Expected version 1 got 0
I20240501 07:58:45.262073 6733 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7]). Expected version 1 got 0
I20240501 07:58:45.262986 11748 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.262998 11733 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.263394 11733 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:45.263427 11748 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:45.266505 11730 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.266898 11730 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:45.269812 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7] alter complete (version 1)
I20240501 07:58:45.276655 9936 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.280269 10706 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.319950 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:45.323316 9250 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.327456 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.328471 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.330578 9644 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.336004 9381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:45.337518 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.345008 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.367318 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:45.375392 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:45.377789 9507 tablet_service.cc:1467] Processing CreateTablet for tablet df79be5e913144a786b208e0a1674951 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.377924 6989 tablet_service.cc:1467] Processing CreateTablet for tablet c93872326cd24c539432897bbd29ed0a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.377996 7121 tablet_service.cc:1467] Processing CreateTablet for tablet c93872326cd24c539432897bbd29ed0a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.378159 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df79be5e913144a786b208e0a1674951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.378258 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93872326cd24c539432897bbd29ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.378321 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93872326cd24c539432897bbd29ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.378501 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 748b249371a4464f88aeb0c4c8558998 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.378768 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b249371a4464f88aeb0c4c8558998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.379617 9382 tablet_service.cc:1467] Processing CreateTablet for tablet c93872326cd24c539432897bbd29ed0a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.379865 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93872326cd24c539432897bbd29ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.380762 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 748b249371a4464f88aeb0c4c8558998 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.380981 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b249371a4464f88aeb0c4c8558998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.381309 9250 tablet_service.cc:1467] Processing CreateTablet for tablet df79be5e913144a786b208e0a1674951 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.381564 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df79be5e913144a786b208e0a1674951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.382699 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 748b249371a4464f88aeb0c4c8558998 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.382930 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b249371a4464f88aeb0c4c8558998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.387887 11766 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:45.388226 11730 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:45.388512 11730 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:45.386919 6859 tablet_service.cc:1467] Processing CreateTablet for tablet df79be5e913144a786b208e0a1674951 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:45.389109 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df79be5e913144a786b208e0a1674951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.391716 11733 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:45.391996 11733 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:45.395107 11770 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:45.396199 11770 tablet_bootstrap.cc:654] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.397652 11770 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:45.397702 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e794f9e6436245e28cd4b76b061464e7] alter complete (version 2)
I20240501 07:58:45.397740 11770 ts_tablet_manager.cc:1367] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:45.398240 11770 raft_consensus.cc:348] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.398347 11770 raft_consensus.cc:374] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.398383 11770 raft_consensus.cc:729] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.398526 11770 consensus_queue.cc:260] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.398841 11770 ts_tablet_manager.cc:1398] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:45.398936 11748 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:45.399971 11748 tablet.cc:1620] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:45.400339 11767 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:45.400592 11768 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:45.400849 11769 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:45.402503 11768 tablet_bootstrap.cc:654] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.403419 11769 tablet_bootstrap.cc:654] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.404891 11767 tablet_bootstrap.cc:654] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.403683 11765 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:45.406219 11765 tablet_bootstrap.cc:654] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.406993 11766 tablet_bootstrap.cc:654] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.406906 11764 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:45.410773 11768 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:45.410877 11768 ts_tablet_manager.cc:1367] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:45.411656 11764 tablet_bootstrap.cc:654] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.412921 11765 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:45.413022 11765 ts_tablet_manager.cc:1367] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:58:45.413846 11769 tablet_bootstrap.cc:492] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:45.413954 11769 ts_tablet_manager.cc:1367] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:45.414518 11769 raft_consensus.cc:348] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.414652 11769 raft_consensus.cc:374] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.414691 11769 raft_consensus.cc:729] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.415205 11767 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:45.415342 11767 ts_tablet_manager.cc:1367] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:45.415370 11768 raft_consensus.cc:348] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.415501 11768 raft_consensus.cc:374] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.415661 11768 raft_consensus.cc:729] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.415792 11769 consensus_queue.cc:260] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.416283 11765 raft_consensus.cc:348] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.416401 11765 raft_consensus.cc:374] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.416447 11765 raft_consensus.cc:729] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.416616 11768 consensus_queue.cc:260] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.417207 11767 raft_consensus.cc:348] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.417368 11767 raft_consensus.cc:374] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.417376 11764 tablet_bootstrap.cc:492] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:45.417459 11767 raft_consensus.cc:729] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.417472 11764 ts_tablet_manager.cc:1367] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:45.417634 11769 ts_tablet_manager.cc:1398] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:45.417658 11767 consensus_queue.cc:260] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.418013 11768 ts_tablet_manager.cc:1398] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:45.418247 11765 consensus_queue.cc:260] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.418635 11768 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:45.419087 11764 raft_consensus.cc:348] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.419202 11764 raft_consensus.cc:374] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.419237 11764 raft_consensus.cc:729] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.419350 11764 consensus_queue.cc:260] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.419699 11764 ts_tablet_manager.cc:1398] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:45.420235 11765 ts_tablet_manager.cc:1398] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:45.420600 11767 ts_tablet_manager.cc:1398] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:45.422890 11766 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:45.424582 11767 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:45.425631 11768 tablet_bootstrap.cc:654] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.425877 11767 tablet_bootstrap.cc:654] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.431126 11766 ts_tablet_manager.cc:1367] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.000s
I20240501 07:58:45.431658 11766 raft_consensus.cc:348] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.431785 11766 raft_consensus.cc:374] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.431823 11766 raft_consensus.cc:729] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.431954 11766 consensus_queue.cc:260] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.432240 11766 ts_tablet_manager.cc:1398] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:45.432381 11767 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:45.432502 11767 ts_tablet_manager.cc:1367] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:45.432945 11767 raft_consensus.cc:348] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.433121 11767 raft_consensus.cc:374] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.433187 11767 raft_consensus.cc:729] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.433393 11767 consensus_queue.cc:260] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.434187 11767 ts_tablet_manager.cc:1398] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:45.437680 11768 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:45.437825 11768 ts_tablet_manager.cc:1367] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:58:45.438515 11768 raft_consensus.cc:348] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.438675 11768 raft_consensus.cc:374] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.438724 11768 raft_consensus.cc:729] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.438846 11768 consensus_queue.cc:260] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.439114 11768 ts_tablet_manager.cc:1398] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:45.444031 10747 raft_consensus.cc:482] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:45.444126 10747 raft_consensus.cc:504] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.444572 10747 leader_election.cc:288] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:45.444892 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df79be5e913144a786b208e0a1674951" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:45.444914 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df79be5e913144a786b208e0a1674951" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:45.445017 9534 raft_consensus.cc:2454] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:45.445044 6877 raft_consensus.cc:2454] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:45.445314 9206 leader_election.cc:302] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:45.445510 10747 raft_consensus.cc:2781] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:45.445574 10747 raft_consensus.cc:482] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:45.445617 10747 raft_consensus.cc:3037] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.446648 10747 raft_consensus.cc:504] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.447046 10747 leader_election.cc:288] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:45.447474 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df79be5e913144a786b208e0a1674951" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:45.447512 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df79be5e913144a786b208e0a1674951" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:45.447589 6877 raft_consensus.cc:3037] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.447616 9534 raft_consensus.cc:3037] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.448585 6877 raft_consensus.cc:2454] T df79be5e913144a786b208e0a1674951 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:45.448760 9534 raft_consensus.cc:2454] T df79be5e913144a786b208e0a1674951 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:45.448940 9206 leader_election.cc:302] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:45.449142 10747 raft_consensus.cc:2781] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.449285 10747 raft_consensus.cc:686] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:45.449420 10747 consensus_queue.cc:237] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.451233 6730 catalog_manager.cc:5484] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:45.474448 6877 consensus_queue.cc:237] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:45.476133 7271 raft_consensus.cc:1261] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:45.476225 6815 consensus_peers.cc:485] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3839146ca1f0452499eb6b5083097d00. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.476326 7138 raft_consensus.cc:1261] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:45.476590 11700 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:45.476763 11436 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:45.478960 11436 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.479399 7138 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.481004 7271 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.483209 6728 catalog_manager.cc:5484] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.500862 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:45.501735 7271 consensus_queue.cc:237] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:45.503432 9403 raft_consensus.cc:1261] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.504022 11817 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.504367 9666 raft_consensus.cc:1261] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:45.504485 7207 consensus_peers.cc:485] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d4ea1beadd542a894f002ae78d3b37f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.504828 11116 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.508304 11748 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:45.510466 11730 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:45.510861 11733 tablet.cc:1722] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:45.514111 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=e794f9e6436245e28cd4b76b061464e7] alter complete (version 3)
I20240501 07:58:45.527079 11116 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:45.527686 9403 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:45.531270 9666 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:45.532009 6737 catalog_manager.cc:5484] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:45.552285 7270 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:45.553624 9534 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:45.553858 7207 consensus_peers.cc:485] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee29c58cdc6544e8933300ad5faecb3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.553887 9666 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.554113 11817 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.554294 11116 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.556492 11116 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:45.558068 9666 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:45.558689 9534 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:45.561667 6730 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:45.573789 11060 raft_consensus.cc:482] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:45.573907 11060 raft_consensus.cc:504] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.574328 11060 leader_election.cc:288] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:45.574762 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93872326cd24c539432897bbd29ed0a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:45.574867 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93872326cd24c539432897bbd29ed0a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:45.574913 7138 raft_consensus.cc:2454] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:45.575070 7010 raft_consensus.cc:2454] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:45.575290 9337 leader_election.cc:302] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:45.575609 11060 raft_consensus.cc:2781] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:45.575676 11060 raft_consensus.cc:482] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:45.575729 11060 raft_consensus.cc:3037] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.576504 11060 raft_consensus.cc:504] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.576826 11060 leader_election.cc:288] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:45.577134 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93872326cd24c539432897bbd29ed0a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:45.577257 7010 raft_consensus.cc:3037] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.578225 7010 raft_consensus.cc:2454] T c93872326cd24c539432897bbd29ed0a P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:45.578485 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93872326cd24c539432897bbd29ed0a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:45.578579 9339 leader_election.cc:302] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:45.578609 7138 raft_consensus.cc:3037] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.578737 11060 raft_consensus.cc:2781] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.578828 11060 raft_consensus.cc:686] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:45.578986 11060 consensus_queue.cc:237] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:45.579717 7138 raft_consensus.cc:2454] T c93872326cd24c539432897bbd29ed0a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:45.580716 6729 catalog_manager.cc:5484] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:45.586066 9266 consensus_queue.cc:237] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:45.587334 9534 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.587550 9666 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:45.587630 9206 consensus_peers.cc:485] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c5e1de6fc394fcb98c64165d74ca317. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.587898 11856 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.588280 11781 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.589955 11856 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.590576 9666 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.591329 9877 raft_consensus.cc:482] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:45.591430 9877 raft_consensus.cc:504] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.591773 9877 leader_election.cc:288] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:45.592231 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b249371a4464f88aeb0c4c8558998" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:45.592337 9266 raft_consensus.cc:2454] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:45.592882 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b249371a4464f88aeb0c4c8558998" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:45.593011 9403 raft_consensus.cc:2454] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:45.593245 6729 catalog_manager.cc:5484] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.593381 9601 leader_election.cc:302] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:45.593518 9877 raft_consensus.cc:2781] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:45.593585 9877 raft_consensus.cc:482] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:45.593628 9877 raft_consensus.cc:3037] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.593978 9534 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:45.594483 9877 raft_consensus.cc:504] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.594784 9877 leader_election.cc:288] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:45.596343 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b249371a4464f88aeb0c4c8558998" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:45.596462 9403 raft_consensus.cc:3037] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.596726 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748b249371a4464f88aeb0c4c8558998" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:45.596827 9266 raft_consensus.cc:3037] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.597440 9403 raft_consensus.cc:2454] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:45.597716 9266 raft_consensus.cc:2454] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:45.598225 9601 leader_election.cc:302] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:45.598456 9877 raft_consensus.cc:2781] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.598604 9877 raft_consensus.cc:686] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:45.598747 9877 consensus_queue.cc:237] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:45.601399 6737 catalog_manager.cc:5484] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:45.616225 7010 consensus_queue.cc:237] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:45.617348 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:45.617496 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:45.617585 6877 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:45.617921 6946 consensus_peers.cc:485] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69ff85c89fa443539b28c0d0cc299cfd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.618166 9936 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:45.618466 7138 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:45.619408 11878 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:45.619869 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 2828bc2999bb4a6a84a9213486981e85
I20240501 07:58:45.620429 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 2828bc2999bb4a6a84a9213486981e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:45 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:45.620445 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 2828bc2999bb4a6a84a9213486981e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:45 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:45.620783 11882 tablet_replica.cc:330] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:45.620858 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 2828bc2999bb4a6a84a9213486981e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:45 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:45.620968 11882 raft_consensus.cc:2229] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:45.621100 11882 raft_consensus.cc:2258] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:45.621166 11881 tablet_replica.cc:330] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:45.621312 11881 raft_consensus.cc:2229] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:45.621527 11881 raft_consensus.cc:2258] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:45.621752 11882 ts_tablet_manager.cc:1875] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:45.621635 9936 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.622009 7138 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.623793 6877 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.624428 11882 ts_tablet_manager.cc:1888] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:45.624550 11882 log.cc:1196] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2828bc2999bb4a6a84a9213486981e85
I20240501 07:58:45.625002 11882 ts_tablet_manager.cc:1909] T 2828bc2999bb4a6a84a9213486981e85 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:45.625890 11881 ts_tablet_manager.cc:1875] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:45.626324 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table_new_name [id=e794f9e6436245e28cd4b76b061464e7]) successfully deleted
I20240501 07:58:45.626413 6737 catalog_manager.cc:5484] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:45.628631 11881 ts_tablet_manager.cc:1888] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:45.628743 11881 log.cc:1196] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2828bc2999bb4a6a84a9213486981e85
I20240501 07:58:45.629340 11881 ts_tablet_manager.cc:1909] T 2828bc2999bb4a6a84a9213486981e85 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:45.630048 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table_new_name [id=e794f9e6436245e28cd4b76b061464e7]) successfully deleted
I20240501 07:58:45.630228 11883 tablet_replica.cc:330] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:45.630368 11883 raft_consensus.cc:2229] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:45.630501 11883 raft_consensus.cc:2258] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:45.631125 11883 ts_tablet_manager.cc:1875] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:45.633425 11883 ts_tablet_manager.cc:1888] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:45.633603 11883 log.cc:1196] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2828bc2999bb4a6a84a9213486981e85
I20240501 07:58:45.633916 11883 ts_tablet_manager.cc:1909] T 2828bc2999bb4a6a84a9213486981e85 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:45.634640 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 2828bc2999bb4a6a84a9213486981e85 (table rebalancer_dynamic_table_new_name [id=e794f9e6436245e28cd4b76b061464e7]) successfully deleted
I20240501 07:58:45.646368 7010 consensus_queue.cc:237] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
W20240501 07:58:45.648259 6944 consensus_peers.cc:485] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c86818cabaf49678cfbcce3480f2645. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.648375 6877 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.648548 9403 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.648730 11878 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.648866 9936 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:45.650275 9936 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:45.650744 9403 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:45.652087 6877 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:45.654047 6728 catalog_manager.cc:5484] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:45.721696 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:45.727162 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]), partition=
I20240501 07:58:45.727483 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf05acfd7dc4754bd863372a5fc0a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.729403 11766 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:45.729590 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]), partition=
I20240501 07:58:45.729854 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf05acfd7dc4754bd863372a5fc0a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.730675 11766 tablet_bootstrap.cc:654] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.732398 11765 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:45.733305 11765 tablet_bootstrap.cc:654] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.733950 11766 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:45.734051 11766 ts_tablet_manager.cc:1367] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:45.734592 11766 raft_consensus.cc:348] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.734723 11766 raft_consensus.cc:374] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.734755 11766 raft_consensus.cc:729] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.734860 11765 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:45.734880 11766 consensus_queue.cc:260] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.734941 11765 ts_tablet_manager.cc:1367] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:45.735219 11766 ts_tablet_manager.cc:1398] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:45.735417 11765 raft_consensus.cc:348] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.735523 11765 raft_consensus.cc:374] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.735563 11765 raft_consensus.cc:729] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.739288 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]), partition=
I20240501 07:58:45.739567 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf05acfd7dc4754bd863372a5fc0a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.740110 11765 consensus_queue.cc:260] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.741132 11765 ts_tablet_manager.cc:1398] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:45.741740 11768 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:45.742719 11768 tablet_bootstrap.cc:654] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:45.744297 11768 tablet_bootstrap.cc:492] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:45.744388 11768 ts_tablet_manager.cc:1367] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:45.744868 11768 raft_consensus.cc:348] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.744983 11768 raft_consensus.cc:374] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:45.745018 11768 raft_consensus.cc:729] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.745133 11768 consensus_queue.cc:260] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.745416 11768 ts_tablet_manager.cc:1398] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:45.752254 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.752439 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:45.752578 9644 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.755851 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.757504 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:45.759066 9381 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.759219 11864 raft_consensus.cc:482] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:45.759320 11864 raft_consensus.cc:504] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.759433 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:45.762445 11864 leader_election.cc:288] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:45.762707 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf05acfd7dc4754bd863372a5fc0a8c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:45.762841 9266 raft_consensus.cc:2454] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:45.763082 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf05acfd7dc4754bd863372a5fc0a8c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:45.763182 9601 leader_election.cc:302] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:45.763217 7010 raft_consensus.cc:2454] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:45.763571 11864 raft_consensus.cc:2781] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:45.763624 11864 raft_consensus.cc:482] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:45.763674 11864 raft_consensus.cc:3037] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.764488 11864 raft_consensus.cc:504] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.764782 11864 leader_election.cc:288] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:45.765195 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf05acfd7dc4754bd863372a5fc0a8c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:45.765296 9266 raft_consensus.cc:3037] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.766222 9266 raft_consensus.cc:2454] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:45.766728 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf05acfd7dc4754bd863372a5fc0a8c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:45.766829 7010 raft_consensus.cc:3037] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:45.767130 9601 leader_election.cc:302] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:45.767757 7010 raft_consensus.cc:2454] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:45.768996 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.770088 11864 raft_consensus.cc:2781] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:45.770190 11864 raft_consensus.cc:686] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:45.770342 11864 consensus_queue.cc:237] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:45.788004 6732 catalog_manager.cc:5484] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:45.863349 11859 consensus_queue.cc:1035] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.867702 11781 consensus_queue.cc:1035] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.874017 11992 ts_tablet_manager.cc:902] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:45.874711 11992 tablet_copy_client.cc:308] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:45.877390 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 3839146ca1f0452499eb6b5083097d00 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:34317)
I20240501 07:58:45.877501 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 3839146ca1f0452499eb6b5083097d00 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:34317: session id = ac6c8bc1b74b483791e262abee7862cf-3839146ca1f0452499eb6b5083097d00
I20240501 07:58:45.878454 6899 tablet_copy_source_session.cc:205] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:45.879213 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3839146ca1f0452499eb6b5083097d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.881544 11992 tablet_copy_client.cc:783] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:45.881783 11992 tablet_copy_client.cc:647] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:45.884088 11992 tablet_copy_client.cc:523] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:45.885329 11992 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:45.898533 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:45.900799 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.903358 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:45.905035 7010 raft_consensus.cc:1261] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.905776 11864 consensus_queue.cc:1035] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.906843 9266 raft_consensus.cc:1261] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:45.907459 11864 consensus_queue.cc:1035] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:45.909812 6731 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]). Expected version 1 got 0
I20240501 07:58:45.911015 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]). Expected version 1 got 0
I20240501 07:58:45.911422 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:45.917291 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:45.921762 11992 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:45.922035 11992 tablet_bootstrap.cc:492] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:45.922207 11992 ts_tablet_manager.cc:1367] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.007s
I20240501 07:58:45.922613 11992 raft_consensus.cc:348] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:45.922716 11992 raft_consensus.cc:729] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.922888 11992 consensus_queue.cc:260] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:45.923285 11992 ts_tablet_manager.cc:1398] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 07:58:45.923426 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:45.924006 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-3839146ca1f0452499eb6b5083097d00 received from {username='slave'} at 127.0.99.135:34317
I20240501 07:58:45.924093 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-3839146ca1f0452499eb6b5083097d00 on tablet 3839146ca1f0452499eb6b5083097d00 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:45.930016 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.935426 9644 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:45.954999 11733 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.955377 11733 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:45.955817 11730 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.956112 11730 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:45.957005 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae]). Expected version 1 got 0
I20240501 07:58:45.958034 9381 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:45.961083 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae] alter complete (version 1)
I20240501 07:58:45.968178 12015 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:45.968842 12014 ts_tablet_manager.cc:902] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:45.969813 12014 tablet_copy_client.cc:308] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:45.968550 12015 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:45.970393 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 8d4ea1beadd542a894f002ae78d3b37f from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:51117)
I20240501 07:58:45.970521 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 8d4ea1beadd542a894f002ae78d3b37f from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:51117: session id = 1a9029bee6a144938f68b363e4cca162-8d4ea1beadd542a894f002ae78d3b37f
I20240501 07:58:45.971518 7292 tablet_copy_source_session.cc:205] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:45.973436 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4ea1beadd542a894f002ae78d3b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.978674 12014 tablet_copy_client.cc:783] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:45.978865 12014 tablet_copy_client.cc:647] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:45.980310 12014 tablet_copy_client.cc:523] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:45.982849 12014 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:45.988006 12014 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:45.988299 12014 tablet_bootstrap.cc:492] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:58:45.988469 12014 ts_tablet_manager.cc:1367] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:58:45.988997 12014 raft_consensus.cc:348] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:45.989117 12014 raft_consensus.cc:729] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:45.989264 12014 consensus_queue.cc:260] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:45.990166 12014 ts_tablet_manager.cc:1398] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:45.991251 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-8d4ea1beadd542a894f002ae78d3b37f received from {username='slave'} at 127.0.99.129:51117
I20240501 07:58:45.991339 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-8d4ea1beadd542a894f002ae78d3b37f on tablet 8d4ea1beadd542a894f002ae78d3b37f with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:45.999850 11060 consensus_queue.cc:1035] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.004390 11060 consensus_queue.cc:1035] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.009577 11992 ts_tablet_manager.cc:902] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:46.010046 11992 tablet_copy_client.cc:308] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:46.010495 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 69ff85c89fa443539b28c0d0cc299cfd from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53657)
I20240501 07:58:46.010617 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 69ff85c89fa443539b28c0d0cc299cfd from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53657: session id = ac6c8bc1b74b483791e262abee7862cf-69ff85c89fa443539b28c0d0cc299cfd
I20240501 07:58:46.011570 7030 tablet_copy_source_session.cc:205] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.012506 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69ff85c89fa443539b28c0d0cc299cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.014689 11992 tablet_copy_client.cc:783] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.014854 11992 tablet_copy_client.cc:647] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.016083 11992 tablet_copy_client.cc:523] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.023097 11992 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.051904 11992 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.052242 11992 tablet_bootstrap.cc:492] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:46.052439 11992 ts_tablet_manager.cc:1367] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.004s
I20240501 07:58:46.052897 11992 raft_consensus.cc:348] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.053027 11992 raft_consensus.cc:729] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.053165 11992 consensus_queue.cc:260] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.053454 11992 ts_tablet_manager.cc:1398] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:46.054172 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-69ff85c89fa443539b28c0d0cc299cfd received from {username='slave'} at 127.0.99.135:53657
I20240501 07:58:46.054256 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-69ff85c89fa443539b28c0d0cc299cfd on tablet 69ff85c89fa443539b28c0d0cc299cfd with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:46.073333 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:46.083796 12015 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.084160 12015 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:46.084134 11730 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.084417 11730 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:46.088086 9644 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.090859 11733 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.091221 11733 tablet.cc:1620] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:46.091421 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d4b91e49ded043cab17c8b96d60a51ae] alter complete (version 2)
I20240501 07:58:46.091856 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.093660 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:46.094673 12081 ts_tablet_manager.cc:902] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:46.096199 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.096593 9507 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:46.098870 12081 tablet_copy_client.cc:308] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:46.099364 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet ee29c58cdc6544e8933300ad5faecb3a from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:49987)
I20240501 07:58:46.099464 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet ee29c58cdc6544e8933300ad5faecb3a from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:49987: session id = 49adb9e99822494196ed22044bfed81f-ee29c58cdc6544e8933300ad5faecb3a
I20240501 07:58:46.100317 7292 tablet_copy_source_session.cc:205] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.108603 12087 ts_tablet_manager.cc:902] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:46.109122 12087 tablet_copy_client.cc:308] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:58:46.109210 9381 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:46.109735 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet 3c5e1de6fc394fcb98c64165d74ca317 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:49879)
I20240501 07:58:46.109853 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet 3c5e1de6fc394fcb98c64165d74ca317 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:49879: session id = fa6964452854415a85ad72f8639768d6-3c5e1de6fc394fcb98c64165d74ca317
I20240501 07:58:46.111253 9291 tablet_copy_source_session.cc:205] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.119374 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.120398 12087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e1de6fc394fcb98c64165d74ca317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.122676 12087 tablet_copy_client.cc:783] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.122819 12087 tablet_copy_client.cc:647] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.127671 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:46.129078 11864 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.131202 12086 ts_tablet_manager.cc:902] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:46.131978 12086 tablet_copy_client.cc:308] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:46.132566 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 0c86818cabaf49678cfbcce3480f2645 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:51345)
I20240501 07:58:46.132658 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 0c86818cabaf49678cfbcce3480f2645 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:51345: session id = 15ab96f6477b4291932409b80af07213-0c86818cabaf49678cfbcce3480f2645
I20240501 07:58:46.133603 7030 tablet_copy_source_session.cc:205] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.134061 9877 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.134909 12086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c86818cabaf49678cfbcce3480f2645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.142707 12086 tablet_copy_client.cc:783] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.142743 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee29c58cdc6544e8933300ad5faecb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.142977 12086 tablet_copy_client.cc:647] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.144539 12086 tablet_copy_client.cc:523] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.145076 12087 tablet_copy_client.cc:523] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.145713 12081 tablet_copy_client.cc:783] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.145918 12081 tablet_copy_client.cc:647] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.146302 12086 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:46.146540 12087 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:46.149121 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:46.149645 12081 tablet_copy_client.cc:523] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.151096 12081 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:46.155478 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 9192ffe3b65341cb9ad6918f1e435408 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.155617 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 9192ffe3b65341cb9ad6918f1e435408 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.155776 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9192ffe3b65341cb9ad6918f1e435408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.155897 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9192ffe3b65341cb9ad6918f1e435408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.158000 12100 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:46.158110 9514 tablet_service.cc:1467] Processing CreateTablet for tablet b40ef631801d42ec954b19f7323dc00a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.158334 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40ef631801d42ec954b19f7323dc00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.159224 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 2b09efd2448a4be7b63636b66e56a030 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.159417 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b09efd2448a4be7b63636b66e56a030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.159680 9381 tablet_service.cc:1467] Processing CreateTablet for tablet b40ef631801d42ec954b19f7323dc00a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.159911 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40ef631801d42ec954b19f7323dc00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.161357 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 9192ffe3b65341cb9ad6918f1e435408 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.161603 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9192ffe3b65341cb9ad6918f1e435408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.163544 9644 tablet_service.cc:1467] Processing CreateTablet for tablet b40ef631801d42ec954b19f7323dc00a (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.163792 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40ef631801d42ec954b19f7323dc00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.163862 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 2b09efd2448a4be7b63636b66e56a030 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.164078 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b09efd2448a4be7b63636b66e56a030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.165959 11768 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:46.165916 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 2b09efd2448a4be7b63636b66e56a030 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.166134 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b09efd2448a4be7b63636b66e56a030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.166766 12100 tablet_bootstrap.cc:654] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.167754 11768 tablet_bootstrap.cc:654] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.167999 11766 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:46.168040 12100 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:46.168129 12100 ts_tablet_manager.cc:1367] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:46.168656 12100 raft_consensus.cc:348] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.168771 12100 raft_consensus.cc:374] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.168807 12100 raft_consensus.cc:729] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.168844 11766 tablet_bootstrap.cc:654] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.168941 12100 consensus_queue.cc:260] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.169231 12100 ts_tablet_manager.cc:1398] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.172382 12102 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:46.172900 12101 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.173409 11766 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:46.173493 11766 ts_tablet_manager.cc:1367] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:46.173933 12102 tablet_bootstrap.cc:654] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.173902 11766 raft_consensus.cc:348] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.173995 11766 raft_consensus.cc:374] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.174021 11766 raft_consensus.cc:729] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.174131 11766 consensus_queue.cc:260] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.174391 11766 ts_tablet_manager.cc:1398] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.174624 11766 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:46.175866 11766 tablet_bootstrap.cc:654] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.178707 12101 tablet_bootstrap.cc:654] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.183751 12087 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.184058 12087 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:46.184259 12087 ts_tablet_manager.cc:1367] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.005s
I20240501 07:58:46.184947 12087 raft_consensus.cc:348] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.185148 12087 raft_consensus.cc:729] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.185245 11768 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:46.185338 11768 ts_tablet_manager.cc:1367] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 07:58:46.185300 12087 consensus_queue.cc:260] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.185639 12087 ts_tablet_manager.cc:1398] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.185741 11768 raft_consensus.cc:348] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.185837 11768 raft_consensus.cc:374] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.185863 11768 raft_consensus.cc:729] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.185967 11768 consensus_queue.cc:260] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.186142 12102 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:46.186216 12102 ts_tablet_manager.cc:1367] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:58:46.186575 12102 raft_consensus.cc:348] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.186658 12102 raft_consensus.cc:374] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.186767 11768 ts_tablet_manager.cc:1398] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.187835 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-3c5e1de6fc394fcb98c64165d74ca317 received from {username='slave'} at 127.0.99.134:49879
I20240501 07:58:46.187956 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session fa6964452854415a85ad72f8639768d6-3c5e1de6fc394fcb98c64165d74ca317 on tablet 3c5e1de6fc394fcb98c64165d74ca317 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:46.186687 12102 raft_consensus.cc:729] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.188668 11766 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:46.188827 11766 ts_tablet_manager.cc:1367] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 07:58:46.188827 12102 consensus_queue.cc:260] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.189116 12102 ts_tablet_manager.cc:1398] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:46.189458 12102 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:46.190826 12102 tablet_bootstrap.cc:654] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.191468 11766 raft_consensus.cc:348] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.192196 12101 tablet_bootstrap.cc:492] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.192304 12101 ts_tablet_manager.cc:1367] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240501 07:58:46.192741 12101 raft_consensus.cc:348] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.192860 12101 raft_consensus.cc:374] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.192903 12101 raft_consensus.cc:729] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.193048 12101 consensus_queue.cc:260] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.193207 12081 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.193445 12101 ts_tablet_manager.cc:1398] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.193495 12081 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:46.193627 12101 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.193661 12081 ts_tablet_manager.cc:1367] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20240501 07:58:46.194113 12081 raft_consensus.cc:348] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.194240 12081 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.194363 12081 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.194669 12081 ts_tablet_manager.cc:1398] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.195681 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-ee29c58cdc6544e8933300ad5faecb3a received from {username='slave'} at 127.0.99.133:49987
I20240501 07:58:46.195783 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 49adb9e99822494196ed22044bfed81f-ee29c58cdc6544e8933300ad5faecb3a on tablet ee29c58cdc6544e8933300ad5faecb3a with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:46.195849 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:46.197134 12086 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.197438 12086 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:58:46.197618 12086 ts_tablet_manager.cc:1367] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.000s
I20240501 07:58:46.198110 12086 raft_consensus.cc:348] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:46.198228 12086 raft_consensus.cc:729] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.198374 12086 consensus_queue.cc:260] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:46.201184 11766 raft_consensus.cc:374] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.201279 11766 raft_consensus.cc:729] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.202095 12086 ts_tablet_manager.cc:1398] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:46.201467 11766 consensus_queue.cc:260] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.202499 11766 ts_tablet_manager.cc:1398] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20240501 07:58:46.203238 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-0c86818cabaf49678cfbcce3480f2645 received from {username='slave'} at 127.0.99.132:51345
I20240501 07:58:46.203327 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 15ab96f6477b4291932409b80af07213-0c86818cabaf49678cfbcce3480f2645 on tablet 0c86818cabaf49678cfbcce3480f2645 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:58:46.204079 12015 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.204604 11730 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.209064 11733 tablet.cc:1722] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.211758 12101 tablet_bootstrap.cc:654] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.212823 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=d4b91e49ded043cab17c8b96d60a51ae] alter complete (version 3)
I20240501 07:58:46.213107 12101 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.213214 12101 ts_tablet_manager.cc:1367] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240501 07:58:46.214304 12101 raft_consensus.cc:348] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.214694 12101 raft_consensus.cc:374] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.214774 12101 raft_consensus.cc:729] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.215553 12101 consensus_queue.cc:260] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.216364 12101 ts_tablet_manager.cc:1398] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:46.216572 12101 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.218016 12101 tablet_bootstrap.cc:654] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.219748 12102 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:46.219839 12102 ts_tablet_manager.cc:1367] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20240501 07:58:46.220356 12102 raft_consensus.cc:348] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.220726 12102 raft_consensus.cc:374] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.220817 12102 raft_consensus.cc:729] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.221030 12102 consensus_queue.cc:260] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.221580 12102 ts_tablet_manager.cc:1398] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:46.222318 12101 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.222402 12101 ts_tablet_manager.cc:1367] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:58:46.222761 12101 raft_consensus.cc:348] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.222833 12101 raft_consensus.cc:374] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.222862 12101 raft_consensus.cc:729] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.222987 12101 consensus_queue.cc:260] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.223294 12101 ts_tablet_manager.cc:1398] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.293452 11060 raft_consensus.cc:482] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:46.293579 11060 raft_consensus.cc:504] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.294082 11060 leader_election.cc:288] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:46.295361 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b09efd2448a4be7b63636b66e56a030" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:46.295547 9665 raft_consensus.cc:2454] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:46.295904 9338 leader_election.cc:302] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:46.296211 11060 raft_consensus.cc:2781] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:46.296267 11060 raft_consensus.cc:482] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:46.296305 11060 raft_consensus.cc:3037] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.298005 11060 raft_consensus.cc:504] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.298416 11060 leader_election.cc:288] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:46.298933 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b09efd2448a4be7b63636b66e56a030" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:46.299116 9665 raft_consensus.cc:3037] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.300177 9665 raft_consensus.cc:2454] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:46.300510 9338 leader_election.cc:302] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:46.300694 11060 raft_consensus.cc:2781] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:46.300855 11060 raft_consensus.cc:686] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:46.300994 11060 consensus_queue.cc:237] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:46.302439 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b09efd2448a4be7b63636b66e56a030" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:46.302439 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b09efd2448a4be7b63636b66e56a030" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:46.302597 9535 raft_consensus.cc:3037] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.303673 9535 raft_consensus.cc:2454] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:46.304767 6736 catalog_manager.cc:5484] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:46.316236 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:46.316359 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:46.318300 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 5cf05acfd7dc4754bd863372a5fc0a8c
I20240501 07:58:46.318909 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:46.318934 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:46.319245 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf05acfd7dc4754bd863372a5fc0a8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:46.319285 12156 tablet_replica.cc:330] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:46.319285 12157 tablet_replica.cc:330] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:46.319461 12156 raft_consensus.cc:2229] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.319474 12157 raft_consensus.cc:2229] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.319541 12158 tablet_replica.cc:330] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:46.319610 12156 raft_consensus.cc:2258] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.319612 12157 raft_consensus.cc:2258] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.319713 12158 raft_consensus.cc:2229] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:46.319927 12158 raft_consensus.cc:2258] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.320158 12157 ts_tablet_manager.cc:1875] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.320463 12158 ts_tablet_manager.cc:1875] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.322388 12158 ts_tablet_manager.cc:1888] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.322388 12157 ts_tablet_manager.cc:1888] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.322464 12157 log.cc:1196] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5cf05acfd7dc4754bd863372a5fc0a8c
I20240501 07:58:46.322552 12156 ts_tablet_manager.cc:1875] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.322573 12158 log.cc:1196] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/5cf05acfd7dc4754bd863372a5fc0a8c
I20240501 07:58:46.322863 12157 ts_tablet_manager.cc:1909] T 5cf05acfd7dc4754bd863372a5fc0a8c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:46.322913 12158 ts_tablet_manager.cc:1909] T 5cf05acfd7dc4754bd863372a5fc0a8c P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:46.323664 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table_new_name [id=d4b91e49ded043cab17c8b96d60a51ae]) successfully deleted
I20240501 07:58:46.323664 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table_new_name [id=d4b91e49ded043cab17c8b96d60a51ae]) successfully deleted
I20240501 07:58:46.324892 12156 ts_tablet_manager.cc:1888] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.324954 12156 log.cc:1196] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5cf05acfd7dc4754bd863372a5fc0a8c
I20240501 07:58:46.325152 12156 ts_tablet_manager.cc:1909] T 5cf05acfd7dc4754bd863372a5fc0a8c P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:46.325758 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 5cf05acfd7dc4754bd863372a5fc0a8c (table rebalancer_dynamic_table_new_name [id=d4b91e49ded043cab17c8b96d60a51ae]) successfully deleted
I20240501 07:58:46.340207 11487 raft_consensus.cc:482] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:46.340340 11487 raft_consensus.cc:504] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.340801 11487 leader_election.cc:288] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:46.341178 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:46.341187 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:46.341320 9266 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:46.341320 7138 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:46.341601 9469 leader_election.cc:302] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:46.341787 11487 raft_consensus.cc:2781] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:46.341841 11487 raft_consensus.cc:482] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:46.341876 11487 raft_consensus.cc:3037] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.342942 11487 raft_consensus.cc:504] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.343410 11487 leader_election.cc:288] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:46.343739 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:46.343753 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:46.343859 7138 raft_consensus.cc:3037] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.343859 9266 raft_consensus.cc:3037] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.344810 9266 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:46.344810 7138 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:46.345098 9470 leader_election.cc:302] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.345302 11487 raft_consensus.cc:2781] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:46.345369 11487 raft_consensus.cc:686] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:46.345508 11487 consensus_queue.cc:237] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:46.347579 6736 catalog_manager.cc:5484] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:46.349289 11864 raft_consensus.cc:482] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:46.349418 11864 raft_consensus.cc:504] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.349824 11864 leader_election.cc:288] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:46.350149 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40ef631801d42ec954b19f7323dc00a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:46.350270 9535 raft_consensus.cc:2454] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:46.350479 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40ef631801d42ec954b19f7323dc00a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:46.350600 9403 raft_consensus.cc:2454] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:46.350821 9602 leader_election.cc:302] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.351100 11864 raft_consensus.cc:2781] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:46.351163 11864 raft_consensus.cc:482] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:46.351217 11864 raft_consensus.cc:3037] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.352030 11864 raft_consensus.cc:504] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.352372 11864 leader_election.cc:288] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:46.352702 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40ef631801d42ec954b19f7323dc00a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:46.352821 9535 raft_consensus.cc:3037] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.353682 9535 raft_consensus.cc:2454] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:46.353967 9602 leader_election.cc:302] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.353972 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40ef631801d42ec954b19f7323dc00a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:46.354074 9403 raft_consensus.cc:3037] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.354300 11864 raft_consensus.cc:2781] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:46.354388 11864 raft_consensus.cc:686] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:46.354511 11864 consensus_queue.cc:237] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.354979 9403 raft_consensus.cc:2454] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:46.356501 6736 catalog_manager.cc:5484] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.374862 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.374931 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.379281 9514 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.379612 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:46.387377 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.389158 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:46.399435 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.399747 9646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.424357 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:46.431195 9535 raft_consensus.cc:1203] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:46.438354 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]), partition=
I20240501 07:58:46.438513 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]), partition=
I20240501 07:58:46.438609 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778211f792f64b7ba04d82b6f5dafe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.438735 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778211f792f64b7ba04d82b6f5dafe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.439393 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]), partition=
I20240501 07:58:46.439596 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778211f792f64b7ba04d82b6f5dafe14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.440623 11766 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:46.441624 11768 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:46.441709 11766 tablet_bootstrap.cc:654] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.443121 11768 tablet_bootstrap.cc:654] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.443863 12101 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.444684 12101 tablet_bootstrap.cc:654] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.444921 11768 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:46.445009 11768 ts_tablet_manager.cc:1367] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:46.445590 11768 raft_consensus.cc:348] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.445806 12101 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.445886 12101 ts_tablet_manager.cc:1367] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:46.446341 12101 raft_consensus.cc:348] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.446444 12101 raft_consensus.cc:374] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.446475 12101 raft_consensus.cc:729] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.446605 12101 consensus_queue.cc:260] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.446897 12101 ts_tablet_manager.cc:1398] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.447119 11768 raft_consensus.cc:374] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.447170 11768 raft_consensus.cc:729] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.447443 11768 consensus_queue.cc:260] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.447728 11768 ts_tablet_manager.cc:1398] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:46.451879 11766 tablet_bootstrap.cc:492] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:46.452005 11766 ts_tablet_manager.cc:1367] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 07:58:46.452484 11766 raft_consensus.cc:348] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.452593 11766 raft_consensus.cc:374] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.452618 11766 raft_consensus.cc:729] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.452721 11766 consensus_queue.cc:260] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.452962 11766 ts_tablet_manager.cc:1398] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.478874 11781 raft_consensus.cc:482] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:46.478991 11781 raft_consensus.cc:504] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.479423 11781 leader_election.cc:288] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:46.479801 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778211f792f64b7ba04d82b6f5dafe14" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:46.479948 9535 raft_consensus.cc:2454] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:46.480243 9207 leader_election.cc:302] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.480434 11781 raft_consensus.cc:2781] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:46.480499 11781 raft_consensus.cc:482] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:46.480546 11781 raft_consensus.cc:3037] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.481595 11781 raft_consensus.cc:504] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.481895 11781 leader_election.cc:288] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:46.482388 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778211f792f64b7ba04d82b6f5dafe14" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:46.482511 9535 raft_consensus.cc:3037] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.483497 9535 raft_consensus.cc:2454] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:46.483909 9207 leader_election.cc:302] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.484086 11781 raft_consensus.cc:2781] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:46.484241 11781 raft_consensus.cc:686] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:46.484413 11781 consensus_queue.cc:237] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.486243 6732 catalog_manager.cc:5484] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.487321 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778211f792f64b7ba04d82b6f5dafe14" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:46.487488 9665 raft_consensus.cc:2454] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:46.487331 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778211f792f64b7ba04d82b6f5dafe14" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:46.487815 9666 raft_consensus.cc:3037] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.488858 9666 raft_consensus.cc:2454] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:46.500914 7271 raft_consensus.cc:1203] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:46.514154 6877 raft_consensus.cc:1203] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:46.561668 9403 raft_consensus.cc:1203] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:46.563606 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.569612 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:46.573594 9238 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:46.576328 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:46.581128 9535 raft_consensus.cc:1203] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:46.582125 9646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.586983 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.592989 9514 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.594530 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:46.602633 9266 raft_consensus.cc:1203] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:46.603091 9666 raft_consensus.cc:1261] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:46.603916 9535 raft_consensus.cc:1261] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:46.604580 12187 consensus_queue.cc:1035] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.605571 12187 consensus_queue.cc:1035] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.608373 6730 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]). Expected version 1 got 0
I20240501 07:58:46.608703 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]). Expected version 1 got 0
I20240501 07:58:46.626657 11733 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:46.627049 11733 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:46.627712 12015 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:46.628000 12015 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:46.631520 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a]). Expected version 1 got 0
I20240501 07:58:46.633147 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a] alter complete (version 1)
I20240501 07:58:46.638046 12233 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:46.638437 12233 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:46.638967 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.677565 6877 consensus_queue.cc:237] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.678869 9535 raft_consensus.cc:1261] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:46.679235 7138 raft_consensus.cc:1261] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:46.679322 12250 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:46.679625 12250 consensus_queue.cc:1035] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:46.680938 11700 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.681360 9535 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.683130 7138 raft_consensus.cc:2932] T 3839146ca1f0452499eb6b5083097d00 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.683810 6729 catalog_manager.cc:5484] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:46.687987 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 3839146ca1f0452499eb6b5083097d00 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:46.688321 12259 tablet_replica.cc:330] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:46.688472 12259 raft_consensus.cc:2229] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.688647 12259 raft_consensus.cc:2258] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.689087 12259 ts_tablet_manager.cc:1875] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:46.691169 12259 ts_tablet_manager.cc:1888] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:46.691256 12259 log.cc:1196] T 3839146ca1f0452499eb6b5083097d00 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3839146ca1f0452499eb6b5083097d00
I20240501 07:58:46.691799 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 3839146ca1f0452499eb6b5083097d00 (table rebalance_test_table_15 [id=16a2fc64af344f04907b5e2ed2ff67f0]) successfully deleted
I20240501 07:58:46.735869 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:46.742115 11733 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.742484 11733 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:46.742707 12233 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.743060 12233 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:46.746920 12015 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:46.747269 12015 tablet.cc:1620] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:46.748188 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=5e85c097aaa544aa8aac2e09ff51ed6a] alter complete (version 2)
I20240501 07:58:46.778635 9238 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:46.783337 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:46.785593 9514 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.786453 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:46.789315 11060 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.795292 9646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:46.803892 11573 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.808072 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.808352 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:46.827627 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:46.838891 11864 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.843860 11864 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.852623 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:46.862077 11733 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.863680 12015 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.867245 12233 tablet.cc:1722] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:46.868127 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=5e85c097aaa544aa8aac2e09ff51ed6a] alter complete (version 3)
I20240501 07:58:46.874598 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:46.881752 9382 tablet_service.cc:1467] Processing CreateTablet for tablet a7603404d6814a3ea8085a7d53772cb7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.881876 9238 tablet_service.cc:1467] Processing CreateTablet for tablet a7603404d6814a3ea8085a7d53772cb7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.882115 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7603404d6814a3ea8085a7d53772cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.882189 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7603404d6814a3ea8085a7d53772cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.883124 9515 tablet_service.cc:1467] Processing CreateTablet for tablet a1d72a9cc6a940b7b15535c8db19c312 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.883402 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d72a9cc6a940b7b15535c8db19c312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.884167 11768 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:46.884423 7252 tablet_service.cc:1467] Processing CreateTablet for tablet a1d72a9cc6a940b7b15535c8db19c312 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.884678 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d72a9cc6a940b7b15535c8db19c312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.885069 11768 tablet_bootstrap.cc:654] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.885107 12322 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:46.885722 6989 tablet_service.cc:1467] Processing CreateTablet for tablet a1d72a9cc6a940b7b15535c8db19c312 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.885944 12322 tablet_bootstrap.cc:654] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.886018 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d72a9cc6a940b7b15535c8db19c312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.886709 12325 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:46.886827 9514 tablet_service.cc:1467] Processing CreateTablet for tablet a7603404d6814a3ea8085a7d53772cb7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.886998 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7603404d6814a3ea8085a7d53772cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.887368 9507 tablet_service.cc:1467] Processing CreateTablet for tablet d30d4d6ddf3a40989b10cf5c10ef7082 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.887562 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30d4d6ddf3a40989b10cf5c10ef7082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.888118 12322 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:46.888200 12322 ts_tablet_manager.cc:1367] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:46.888407 12101 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.888882 12322 raft_consensus.cc:348] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.888989 12322 raft_consensus.cc:374] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.889020 12322 raft_consensus.cc:729] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.889199 12101 tablet_bootstrap.cc:654] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.889199 12322 consensus_queue.cc:260] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.890496 9382 tablet_service.cc:1467] Processing CreateTablet for tablet d30d4d6ddf3a40989b10cf5c10ef7082 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.890745 12322 ts_tablet_manager.cc:1398] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:46.891077 11768 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:46.891191 11768 ts_tablet_manager.cc:1367] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 07:58:46.891216 12325 tablet_bootstrap.cc:654] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.891525 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30d4d6ddf3a40989b10cf5c10ef7082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.891680 11768 raft_consensus.cc:348] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.891780 11768 raft_consensus.cc:374] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.891815 11768 raft_consensus.cc:729] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.891819 12326 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:46.891927 11768 consensus_queue.cc:260] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.892175 11768 ts_tablet_manager.cc:1398] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.892565 12325 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:46.892642 12325 ts_tablet_manager.cc:1367] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:46.892714 12101 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.892784 12101 ts_tablet_manager.cc:1367] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:46.893056 12325 raft_consensus.cc:348] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.893180 12325 raft_consensus.cc:374] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.893213 12325 raft_consensus.cc:729] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.893164 12101 raft_consensus.cc:348] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.893258 12101 raft_consensus.cc:374] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.893290 12101 raft_consensus.cc:729] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.893330 12325 consensus_queue.cc:260] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.893400 12101 consensus_queue.cc:260] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.893622 12325 ts_tablet_manager.cc:1398] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.893672 12101 ts_tablet_manager.cc:1398] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.893811 12101 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.894526 12322 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:46.894801 12101 tablet_bootstrap.cc:654] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.895473 12322 tablet_bootstrap.cc:654] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.898113 9238 tablet_service.cc:1467] Processing CreateTablet for tablet d30d4d6ddf3a40989b10cf5c10ef7082 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:46.898393 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30d4d6ddf3a40989b10cf5c10ef7082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.900411 11768 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:46.900900 12326 tablet_bootstrap.cc:654] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.901036 12322 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:46.901127 12322 ts_tablet_manager.cc:1367] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:46.901490 11768 tablet_bootstrap.cc:654] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.901599 12322 raft_consensus.cc:348] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.901717 12322 raft_consensus.cc:374] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.901747 12322 raft_consensus.cc:729] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.901903 12326 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:46.901870 12322 consensus_queue.cc:260] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.901993 12326 ts_tablet_manager.cc:1367] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:58:46.902132 12322 ts_tablet_manager.cc:1398] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:46.902524 12326 raft_consensus.cc:348] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.902628 12326 raft_consensus.cc:374] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.902659 12326 raft_consensus.cc:729] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.902720 11768 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:46.902788 11768 ts_tablet_manager.cc:1367] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:46.902809 12326 consensus_queue.cc:260] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.902928 11487 consensus_queue.cc:1035] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.903290 11768 raft_consensus.cc:348] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.903461 11768 raft_consensus.cc:374] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.903537 11768 raft_consensus.cc:729] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.903692 12101 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.903719 11768 consensus_queue.cc:260] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.903786 12101 ts_tablet_manager.cc:1367] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 07:58:46.904206 12101 raft_consensus.cc:348] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.904309 12101 raft_consensus.cc:374] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.904338 12101 raft_consensus.cc:729] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.904453 12101 consensus_queue.cc:260] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.904522 11768 ts_tablet_manager.cc:1398] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:46.904696 12101 ts_tablet_manager.cc:1398] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:46.904846 12101 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:46.905736 12101 tablet_bootstrap.cc:654] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:46.906663 12326 ts_tablet_manager.cc:1398] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:46.907516 12101 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:46.907613 12101 ts_tablet_manager.cc:1367] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:46.908025 12101 raft_consensus.cc:348] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.908140 12101 raft_consensus.cc:374] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:46.908174 12101 raft_consensus.cc:729] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:46.908293 12101 consensus_queue.cc:260] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:46.908586 12101 ts_tablet_manager.cc:1398] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:46.910676 11487 consensus_queue.cc:1035] T 9192ffe3b65341cb9ad6918f1e435408 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:46.915680 7271 consensus_queue.cc:237] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:46.917009 7138 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:46.917017 9666 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:46.917414 11116 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:46.917596 11820 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:46.919438 11820 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:46.919983 9666 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:46.920004 7138 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:46.922781 6730 catalog_manager.cc:5484] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:46.925582 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7f613e16c245de83fde8c277b8d350 with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:46.925966 12350 tablet_replica.cc:330] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:46.926213 12350 raft_consensus.cc:2229] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.926383 12350 raft_consensus.cc:2258] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.926934 12350 ts_tablet_manager.cc:1875] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:46.927856 11487 raft_consensus.cc:482] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:46.927959 11487 raft_consensus.cc:504] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.928382 11487 leader_election.cc:288] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:46.928759 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:46.928846 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:46.928898 7271 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:46.928983 7010 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:46.929325 9469 leader_election.cc:302] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.929559 11487 raft_consensus.cc:2781] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:46.929615 11487 raft_consensus.cc:482] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:46.929673 11487 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.930207 12350 ts_tablet_manager.cc:1888] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:46.930279 12350 log.cc:1196] T 4e7f613e16c245de83fde8c277b8d350 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4e7f613e16c245de83fde8c277b8d350
I20240501 07:58:46.930636 11487 raft_consensus.cc:504] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.930905 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 4e7f613e16c245de83fde8c277b8d350 (table rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]) successfully deleted
I20240501 07:58:46.931026 11487 leader_election.cc:288] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:46.931497 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:46.931609 7271 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.932274 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:46.932397 7010 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:46.932479 7271 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:46.933020 9469 leader_election.cc:302] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:46.933209 11487 raft_consensus.cc:2781] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:46.933297 11487 raft_consensus.cc:686] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:46.933421 7010 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:46.933423 11487 consensus_queue.cc:237] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:46.935587 6734 catalog_manager.cc:5484] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:46.972164 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:46.972319 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:46.974514 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 778211f792f64b7ba04d82b6f5dafe14
I20240501 07:58:46.975080 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:46.975179 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:46.975330 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 778211f792f64b7ba04d82b6f5dafe14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:46 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:46.975663 12390 tablet_replica.cc:330] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:46.975883 12391 tablet_replica.cc:330] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:46.976081 12390 raft_consensus.cc:2229] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.976980 12391 raft_consensus.cc:2229] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:46.979857 12390 raft_consensus.cc:2258] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.980535 12391 raft_consensus.cc:2258] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.981973 12390 ts_tablet_manager.cc:1875] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.984090 12389 tablet_replica.cc:330] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:46.985340 12390 ts_tablet_manager.cc:1888] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.985563 12389 raft_consensus.cc:2229] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:46.985816 12389 raft_consensus.cc:2258] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:46.986084 12391 ts_tablet_manager.cc:1875] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.986487 12389 ts_tablet_manager.cc:1875] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:46.987481 12390 log.cc:1196] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/778211f792f64b7ba04d82b6f5dafe14
I20240501 07:58:46.987871 12390 ts_tablet_manager.cc:1909] T 778211f792f64b7ba04d82b6f5dafe14 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:46.989176 12391 ts_tablet_manager.cc:1888] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.989300 12391 log.cc:1196] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/778211f792f64b7ba04d82b6f5dafe14
I20240501 07:58:46.989660 12391 ts_tablet_manager.cc:1909] T 778211f792f64b7ba04d82b6f5dafe14 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:46.990298 12389 ts_tablet_manager.cc:1888] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:46.990319 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table_new_name [id=5e85c097aaa544aa8aac2e09ff51ed6a]) successfully deleted
I20240501 07:58:46.990459 12389 log.cc:1196] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/778211f792f64b7ba04d82b6f5dafe14
I20240501 07:58:46.990792 12389 ts_tablet_manager.cc:1909] T 778211f792f64b7ba04d82b6f5dafe14 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:46.991489 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table_new_name [id=5e85c097aaa544aa8aac2e09ff51ed6a]) successfully deleted
I20240501 07:58:46.991634 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 778211f792f64b7ba04d82b6f5dafe14 (table rebalancer_dynamic_table_new_name [id=5e85c097aaa544aa8aac2e09ff51ed6a]) successfully deleted
I20240501 07:58:47.039469 11763 raft_consensus.cc:482] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.039583 11763 raft_consensus.cc:504] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.039959 11763 leader_election.cc:288] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.040342 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:47.040339 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:47.040484 9403 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:47.040484 9266 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:47.040740 9470 leader_election.cc:302] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:47.040940 11763 raft_consensus.cc:2781] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.041011 11763 raft_consensus.cc:482] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.041056 11763 raft_consensus.cc:3037] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.041877 11763 raft_consensus.cc:504] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.042150 11763 leader_election.cc:288] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.042475 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:47.042537 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.042584 9402 raft_consensus.cc:3037] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.042657 9266 raft_consensus.cc:3037] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.043394 9266 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:47.043439 9402 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:47.043535 11763 raft_consensus.cc:482] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.043623 11763 raft_consensus.cc:504] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.043812 9470 leader_election.cc:302] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:47.043947 11763 leader_election.cc:288] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.044041 11763 raft_consensus.cc:2781] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.044137 11763 raft_consensus.cc:686] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:47.044263 11763 consensus_queue.cc:237] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.044636 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:47.044728 9266 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:47.044991 9470 leader_election.cc:302] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:47.045143 11487 raft_consensus.cc:2781] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.045199 11487 raft_consensus.cc:482] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.045246 11487 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.045854 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:47.045969 9402 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:47.046207 11487 raft_consensus.cc:504] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.046512 11487 leader_election.cc:288] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.046783 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.046819 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:47.046864 9266 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.046919 9402 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.047245 6728 catalog_manager.cc:5484] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:47.047542 9266 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:47.047811 9470 leader_election.cc:302] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:47.047884 9402 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:47.047983 11487 raft_consensus.cc:2781] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.048069 11487 raft_consensus.cc:686] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:47.048213 11487 consensus_queue.cc:237] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:47.051409 6728 catalog_manager.cc:5484] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:47.073101 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.073565 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.076598 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:47.084296 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:47.087881 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]), partition=
I20240501 07:58:47.087908 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]), partition=
I20240501 07:58:47.088166 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3a86bc1be436bab05b39ad3141cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.088179 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3a86bc1be436bab05b39ad3141cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.089984 12326 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:47.090193 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]), partition=
I20240501 07:58:47.090368 12322 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:47.090426 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe3a86bc1be436bab05b39ad3141cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.090904 12326 tablet_bootstrap.cc:654] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.091441 12322 tablet_bootstrap.cc:654] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.092686 12322 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:47.092780 12322 ts_tablet_manager.cc:1367] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:47.092705 12407 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:47.093231 12322 raft_consensus.cc:348] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.093355 12322 raft_consensus.cc:374] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.093389 12322 raft_consensus.cc:729] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.093506 12322 consensus_queue.cc:260] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.093776 12322 ts_tablet_manager.cc:1398] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.093816 12407 tablet_bootstrap.cc:654] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.096935 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:47.105109 9507 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.107095 12326 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:47.107205 12326 ts_tablet_manager.cc:1367] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240501 07:58:47.107697 12326 raft_consensus.cc:348] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.107823 12326 raft_consensus.cc:374] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.107859 12326 raft_consensus.cc:729] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.107985 12326 consensus_queue.cc:260] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.108244 12326 ts_tablet_manager.cc:1398] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:47.110931 12407 tablet_bootstrap.cc:492] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:47.111106 12407 ts_tablet_manager.cc:1367] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 07:58:47.111761 12407 raft_consensus.cc:348] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.111935 12407 raft_consensus.cc:374] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.111999 12407 raft_consensus.cc:729] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.112198 12407 consensus_queue.cc:260] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.113128 12407 ts_tablet_manager.cc:1398] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:47.116052 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.123631 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:47.123734 9646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:47.186237 7010 consensus_queue.cc:237] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.187530 7271 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:47.187707 9402 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:47.188115 11899 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:47.188638 11899 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:47.190240 12172 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:47.190631 7271 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:47.190901 9402 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:47.194171 6728 catalog_manager.cc:5484] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:47.198812 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 651beb63e03d40bca88f01ebed416754 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:47.199371 12433 tablet_replica.cc:330] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:47.199599 12433 raft_consensus.cc:2229] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:47.199842 12433 raft_consensus.cc:2258] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.200461 12433 ts_tablet_manager.cc:1875] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:47.202589 12433 ts_tablet_manager.cc:1888] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:47.202652 12433 log.cc:1196] T 651beb63e03d40bca88f01ebed416754 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/651beb63e03d40bca88f01ebed416754
I20240501 07:58:47.203137 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 651beb63e03d40bca88f01ebed416754 (table rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]) successfully deleted
I20240501 07:58:47.313182 12080 raft_consensus.cc:482] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.313345 12080 raft_consensus.cc:504] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.313891 12080 leader_election.cc:288] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.314352 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3a86bc1be436bab05b39ad3141cb9" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:47.314468 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3a86bc1be436bab05b39ad3141cb9" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:47.314605 7008 raft_consensus.cc:2454] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.314618 9402 raft_consensus.cc:2454] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.314913 9602 leader_election.cc:302] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:47.315165 12080 raft_consensus.cc:2781] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.315232 12080 raft_consensus.cc:482] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.315284 12080 raft_consensus.cc:3037] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.316229 12080 raft_consensus.cc:504] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.316524 12080 leader_election.cc:288] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.316838 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3a86bc1be436bab05b39ad3141cb9" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:47.316953 9402 raft_consensus.cc:3037] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.317937 9402 raft_consensus.cc:2454] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:47.318217 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe3a86bc1be436bab05b39ad3141cb9" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:47.318287 9601 leader_election.cc:302] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:47.318341 7008 raft_consensus.cc:3037] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.318477 12080 raft_consensus.cc:2781] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.318574 12080 raft_consensus.cc:686] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:47.318712 12080 consensus_queue.cc:237] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:47.319430 7008 raft_consensus.cc:2454] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:47.320677 6736 catalog_manager.cc:5484] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:47.333321 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.337299 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:47.339304 9646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:47.348454 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:47.351086 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.362265 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.366178 9507 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.377310 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:47.417721 7271 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a"
dest_uuid: "15ab96f6477b4291932409b80af07213"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57306
I20240501 07:58:47.417901 7271 raft_consensus.cc:595] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:47.442818 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:47.447937 7008 raft_consensus.cc:1261] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:47.447937 9402 raft_consensus.cc:1261] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:47.448365 12080 consensus_queue.cc:1035] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.448702 12080 consensus_queue.cc:1035] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.451383 6736 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]). Expected version 1 got 0
I20240501 07:58:47.452463 6730 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]). Expected version 1 got 0
I20240501 07:58:47.453132 12525 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:47.453630 12526 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:47.455261 12525 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:47.455246 12527 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:47.455785 12527 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:47.456739 12526 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:47.456681 6736 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32]). Expected version 1 got 0
I20240501 07:58:47.461091 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32] alter complete (version 1)
I20240501 07:58:47.478694 11487 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.482084 11763 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.519049 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.519146 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.521127 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:47.521359 9238 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:47.522320 9646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:47.533627 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:47.538901 11763 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.545696 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.555900 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:47.563172 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 7614555b22a847a2824f48d92611edda (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.563198 9507 tablet_service.cc:1467] Processing CreateTablet for tablet d38445516d1e49d88ef9707246d70115 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.563524 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7614555b22a847a2824f48d92611edda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.563527 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38445516d1e49d88ef9707246d70115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.563779 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 7614555b22a847a2824f48d92611edda (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.564036 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7614555b22a847a2824f48d92611edda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.566202 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 7614555b22a847a2824f48d92611edda (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.566283 9644 tablet_service.cc:1467] Processing CreateTablet for tablet d38445516d1e49d88ef9707246d70115 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.566473 12565 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:47.566507 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38445516d1e49d88ef9707246d70115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.566514 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7614555b22a847a2824f48d92611edda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.567288 9250 tablet_service.cc:1467] Processing CreateTablet for tablet d38445516d1e49d88ef9707246d70115 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.567456 12565 tablet_bootstrap.cc:654] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.567531 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38445516d1e49d88ef9707246d70115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.568189 12564 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:47.569016 9382 tablet_service.cc:1467] Processing CreateTablet for tablet bb0a970a4e624499aa3d51e1bc961cfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.569151 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:47.569300 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0a970a4e624499aa3d51e1bc961cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.569638 9251 tablet_service.cc:1467] Processing CreateTablet for tablet bb0a970a4e624499aa3d51e1bc961cfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.569861 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0a970a4e624499aa3d51e1bc961cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.571135 12407 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:47.571444 12564 tablet_bootstrap.cc:654] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.571991 12407 tablet_bootstrap.cc:654] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.571947 7252 tablet_service.cc:1467] Processing CreateTablet for tablet bb0a970a4e624499aa3d51e1bc961cfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:47.572216 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0a970a4e624499aa3d51e1bc961cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.573236 12407 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:47.573330 12407 ts_tablet_manager.cc:1367] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:47.573792 12407 raft_consensus.cc:348] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.573932 12407 raft_consensus.cc:374] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.573973 12407 raft_consensus.cc:729] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.574115 12407 consensus_queue.cc:260] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.574543 11763 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.574662 12407 ts_tablet_manager.cc:1398] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.577088 12322 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:47.577957 12407 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:47.578115 12322 tablet_bootstrap.cc:654] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.579248 12407 tablet_bootstrap.cc:654] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.579407 9507 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.580485 12407 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:47.580557 12407 ts_tablet_manager.cc:1367] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:47.580945 12407 raft_consensus.cc:348] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.581045 12407 raft_consensus.cc:374] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.581072 12407 raft_consensus.cc:729] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.581192 12407 consensus_queue.cc:260] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.581420 12407 ts_tablet_manager.cc:1398] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.582645 12525 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:47.582779 12567 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:47.582943 12525 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:47.586382 12567 tablet_bootstrap.cc:654] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.586907 12322 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:47.587028 12322 ts_tablet_manager.cc:1367] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:47.587538 12526 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:47.587518 12322 raft_consensus.cc:348] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.587643 12322 raft_consensus.cc:374] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.587679 12322 raft_consensus.cc:729] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.587821 12526 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:47.587792 12322 consensus_queue.cc:260] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.588084 12322 ts_tablet_manager.cc:1398] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.588510 12527 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:47.588830 12527 tablet.cc:1620] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:47.593463 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=62b8777fcb7f4679b433d70433149f32] alter complete (version 2)
I20240501 07:58:47.596817 12567 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:47.596971 12567 ts_tablet_manager.cc:1367] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:58:47.597741 12567 raft_consensus.cc:348] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.597925 12567 raft_consensus.cc:374] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.598049 12567 raft_consensus.cc:729] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.598560 12567 consensus_queue.cc:260] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.598987 12567 ts_tablet_manager.cc:1398] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:47.604629 12564 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:47.604734 12564 ts_tablet_manager.cc:1367] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240501 07:58:47.605209 12564 raft_consensus.cc:348] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.605319 12564 raft_consensus.cc:374] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.605343 12564 raft_consensus.cc:729] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.605443 12564 consensus_queue.cc:260] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.605661 12564 ts_tablet_manager.cc:1398] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.610204 12565 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:47.610297 12565 ts_tablet_manager.cc:1367] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240501 07:58:47.610762 12565 raft_consensus.cc:348] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.610865 12565 raft_consensus.cc:374] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.610891 12565 raft_consensus.cc:729] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.611027 12565 consensus_queue.cc:260] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.611306 12565 ts_tablet_manager.cc:1398] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.611691 12565 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:47.612735 12565 tablet_bootstrap.cc:654] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.620002 12565 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:47.620179 12565 ts_tablet_manager.cc:1367] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 07:58:47.621028 12565 raft_consensus.cc:348] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.621563 12570 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:47.621647 12565 raft_consensus.cc:374] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.621726 12565 raft_consensus.cc:729] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.621883 12565 consensus_queue.cc:260] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.622676 12570 tablet_bootstrap.cc:654] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.622984 12565 ts_tablet_manager.cc:1398] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:47.623216 12565 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:47.624663 12570 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:47.625054 12565 tablet_bootstrap.cc:654] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.626045 12570 ts_tablet_manager.cc:1367] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:47.626571 12570 raft_consensus.cc:348] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.626729 12570 raft_consensus.cc:374] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.626796 12570 raft_consensus.cc:729] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.626971 12570 consensus_queue.cc:260] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.628224 12570 ts_tablet_manager.cc:1398] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:47.630263 12565 tablet_bootstrap.cc:492] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:47.630398 12565 ts_tablet_manager.cc:1367] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 07:58:47.630774 12565 raft_consensus.cc:348] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.630889 12565 raft_consensus.cc:374] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.630926 12565 raft_consensus.cc:729] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.631067 12565 consensus_queue.cc:260] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.631373 12565 ts_tablet_manager.cc:1398] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:47.645510 11487 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.648914 11763 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:47.674330 11781 raft_consensus.cc:482] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.674451 11781 raft_consensus.cc:504] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.674924 11781 leader_election.cc:288] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.675554 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0a970a4e624499aa3d51e1bc961cfd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:47.675554 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0a970a4e624499aa3d51e1bc961cfd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:47.675694 7271 raft_consensus.cc:2454] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:47.675694 9402 raft_consensus.cc:2454] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:47.676000 9206 leader_election.cc:302] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:47.676301 11781 raft_consensus.cc:2781] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.676367 11781 raft_consensus.cc:482] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.676403 11781 raft_consensus.cc:3037] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.677239 11781 raft_consensus.cc:504] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.677559 11781 leader_election.cc:288] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:47.677861 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0a970a4e624499aa3d51e1bc961cfd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:47.678009 9402 raft_consensus.cc:3037] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.678910 9402 raft_consensus.cc:2454] T bb0a970a4e624499aa3d51e1bc961cfd P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:47.679150 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0a970a4e624499aa3d51e1bc961cfd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:47.679212 9206 leader_election.cc:302] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:47.679281 7271 raft_consensus.cc:3037] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.679348 11781 raft_consensus.cc:2781] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.679512 11781 raft_consensus.cc:686] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:47.679654 11781 consensus_queue.cc:237] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.680137 7271 raft_consensus.cc:2454] T bb0a970a4e624499aa3d51e1bc961cfd P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:47.681840 6733 catalog_manager.cc:5484] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:47.690474 12630 raft_consensus.cc:979] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: : Instructing follower ac6c8bc1b74b483791e262abee7862cf to start an election
I20240501 07:58:47.690656 11820 raft_consensus.cc:1067] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Signalling peer ac6c8bc1b74b483791e262abee7862cf to start an election
I20240501 07:58:47.691340 12574 raft_consensus.cc:482] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.691438 12574 raft_consensus.cc:504] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.691814 12574 leader_election.cc:288] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.692597 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38445516d1e49d88ef9707246d70115" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:47.692618 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38445516d1e49d88ef9707246d70115" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:47.692723 9534 raft_consensus.cc:2454] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.692744 9266 raft_consensus.cc:2454] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.693024 9601 leader_election.cc:302] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:47.693185 12574 raft_consensus.cc:2781] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.693250 12574 raft_consensus.cc:482] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.693251 9535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a"
dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
from {username='slave'} at 127.0.99.132:48037
I20240501 07:58:47.693302 12574 raft_consensus.cc:3037] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.693369 9535 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:47.693408 9535 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.694460 12574 raft_consensus.cc:504] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.694463 9535 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.694811 12574 leader_election.cc:288] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.694897 9535 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 2 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.695451 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38445516d1e49d88ef9707246d70115" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.695573 9266 raft_consensus.cc:3037] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.696414 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:47.696519 9666 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.696578 9266 raft_consensus.cc:2454] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:47.696961 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.697057 9271 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.697526 9666 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 2.
I20240501 07:58:47.697867 9271 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 2.
I20240501 07:58:47.698259 9470 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:47.698428 11763 raft_consensus.cc:2781] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:47.698513 11763 raft_consensus.cc:686] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:47.698513 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:47.698643 11763 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.700261 9601 leader_election.cc:302] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:47.700446 12574 raft_consensus.cc:2781] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.700533 12574 raft_consensus.cc:686] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:47.700676 12574 consensus_queue.cc:237] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:47.701017 6728 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 1 to 2, leader changed from 15ab96f6477b4291932409b80af07213 (127.0.99.132) to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 2 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:47.701150 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38445516d1e49d88ef9707246d70115" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:47.701252 9534 raft_consensus.cc:3037] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.702246 9534 raft_consensus.cc:2454] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:47.702764 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:47.702888 7271 raft_consensus.cc:3032] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:47.702929 7271 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:47.703805 7271 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:47.704027 7271 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.704691 6732 catalog_manager.cc:5484] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:47.705053 7271 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 2.
I20240501 07:58:47.707559 12525 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:47.708638 12527 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:47.709231 12526 tablet.cc:1722] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:47.718886 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=62b8777fcb7f4679b433d70433149f32] alter complete (version 3)
I20240501 07:58:47.821712 6728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:47.821872 6728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:47.823976 6728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 3fe3a86bc1be436bab05b39ad3141cb9
I20240501 07:58:47.824506 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:47 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:47.824577 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:47 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:47.824659 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe3a86bc1be436bab05b39ad3141cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:47 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:47.824890 12649 tablet_replica.cc:330] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:47.824923 12650 tablet_replica.cc:330] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:47.825032 12651 tablet_replica.cc:330] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:47.825062 12650 raft_consensus.cc:2229] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:47.825062 12649 raft_consensus.cc:2229] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:47.825191 12649 raft_consensus.cc:2258] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.825188 12651 raft_consensus.cc:2229] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:47.825428 12650 raft_consensus.cc:2258] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.825525 12651 raft_consensus.cc:2258] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.825829 12649 ts_tablet_manager.cc:1875] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:47.826020 12650 ts_tablet_manager.cc:1875] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:47.826170 12651 ts_tablet_manager.cc:1875] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:47.828308 12651 ts_tablet_manager.cc:1888] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:47.828310 12649 ts_tablet_manager.cc:1888] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:47.828326 12650 ts_tablet_manager.cc:1888] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:47.828387 12650 log.cc:1196] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/3fe3a86bc1be436bab05b39ad3141cb9
I20240501 07:58:47.828389 12651 log.cc:1196] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/3fe3a86bc1be436bab05b39ad3141cb9
I20240501 07:58:47.828401 12649 log.cc:1196] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fe3a86bc1be436bab05b39ad3141cb9
I20240501 07:58:47.828644 12651 ts_tablet_manager.cc:1909] T 3fe3a86bc1be436bab05b39ad3141cb9 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:47.828698 12650 ts_tablet_manager.cc:1909] T 3fe3a86bc1be436bab05b39ad3141cb9 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:47.828747 12649 ts_tablet_manager.cc:1909] T 3fe3a86bc1be436bab05b39ad3141cb9 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:47.829349 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table_new_name [id=62b8777fcb7f4679b433d70433149f32]) successfully deleted
I20240501 07:58:47.829349 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table_new_name [id=62b8777fcb7f4679b433d70433149f32]) successfully deleted
I20240501 07:58:47.829550 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 3fe3a86bc1be436bab05b39ad3141cb9 (table rebalancer_dynamic_table_new_name [id=62b8777fcb7f4679b433d70433149f32]) successfully deleted
I20240501 07:58:47.854804 12574 raft_consensus.cc:482] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.854926 12574 raft_consensus.cc:504] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.855425 12574 leader_election.cc:288] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:47.855823 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7614555b22a847a2824f48d92611edda" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:47.855983 9271 raft_consensus.cc:2454] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.856304 9601 leader_election.cc:302] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:47.856478 12574 raft_consensus.cc:2781] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.856532 12574 raft_consensus.cc:482] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.856575 12574 raft_consensus.cc:3037] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.857635 12574 raft_consensus.cc:504] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.858027 12574 leader_election.cc:288] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:47.858356 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7614555b22a847a2824f48d92611edda" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.858475 9271 raft_consensus.cc:3037] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.859426 9271 raft_consensus.cc:2454] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:47.859766 9601 leader_election.cc:302] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:47.859941 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7614555b22a847a2824f48d92611edda" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:47.859973 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7614555b22a847a2824f48d92611edda" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:47.860060 6876 raft_consensus.cc:2454] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:47.860235 12574 raft_consensus.cc:2781] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.860332 12574 raft_consensus.cc:686] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:47.860463 12574 consensus_queue.cc:237] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:47.862232 6732 catalog_manager.cc:5484] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:47.889923 7252 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.890141 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:47.890455 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:47.895879 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:47.896209 9644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:47.899628 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.908582 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:47.908861 9507 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:47.931038 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:47.938453 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a3f9612e74665b0c8440918404350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d]), partition=
I20240501 07:58:47.938577 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a3f9612e74665b0c8440918404350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d]), partition=
I20240501 07:58:47.938716 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a3f9612e74665b0c8440918404350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.938812 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a3f9612e74665b0c8440918404350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.940805 12565 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:47.940912 12564 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:47.941644 12565 tablet_bootstrap.cc:654] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.941758 12564 tablet_bootstrap.cc:654] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.942633 12565 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:47.942713 12565 ts_tablet_manager.cc:1367] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:47.943172 12565 raft_consensus.cc:348] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.943285 12565 raft_consensus.cc:374] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.943311 12565 raft_consensus.cc:729] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.943370 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9a3f9612e74665b0c8440918404350 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d]), partition=
I20240501 07:58:47.943420 12565 consensus_queue.cc:260] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.943607 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9a3f9612e74665b0c8440918404350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:47.943677 12565 ts_tablet_manager.cc:1398] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.945009 12564 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:47.945113 12564 ts_tablet_manager.cc:1367] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:47.945570 12564 raft_consensus.cc:348] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.945673 12564 raft_consensus.cc:374] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.945703 12564 raft_consensus.cc:729] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.945833 12564 consensus_queue.cc:260] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.946106 12564 ts_tablet_manager.cc:1398] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.950405 12668 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:47.951287 12668 tablet_bootstrap.cc:654] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:47.952291 12668 tablet_bootstrap.cc:492] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:47.952368 12668 ts_tablet_manager.cc:1367] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:47.952780 12668 raft_consensus.cc:348] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.952874 12668 raft_consensus.cc:374] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:47.952909 12668 raft_consensus.cc:729] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:47.953037 12668 consensus_queue.cc:260] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.953411 12668 ts_tablet_manager.cc:1398] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:47.978121 7008 consensus_queue.cc:237] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:47.979552 7271 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:47.979552 9403 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:47.980024 12648 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:47.981022 12648 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:47.982633 12648 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:47.982983 9402 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:47.985687 7271 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:47.986017 6732 catalog_manager.cc:5484] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:47.989579 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c86818cabaf49678cfbcce3480f2645 with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:47.990208 12690 tablet_replica.cc:330] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:47.990355 12690 raft_consensus.cc:2229] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:47.990500 12690 raft_consensus.cc:2258] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.991117 12690 ts_tablet_manager.cc:1875] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:47.991608 12253 raft_consensus.cc:482] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:47.991708 12253 raft_consensus.cc:504] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.992097 12253 leader_election.cc:288] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.992570 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a3f9612e74665b0c8440918404350" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:47.992648 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a3f9612e74665b0c8440918404350" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:47.992702 9271 raft_consensus.cc:2454] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:47.992770 9535 raft_consensus.cc:2454] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:47.993036 7076 leader_election.cc:302] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:47.993227 12253 raft_consensus.cc:2781] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:47.993301 12253 raft_consensus.cc:482] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:47.993337 12253 raft_consensus.cc:3037] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.993505 12690 ts_tablet_manager.cc:1888] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:47.993615 12690 log.cc:1196] T 0c86818cabaf49678cfbcce3480f2645 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0c86818cabaf49678cfbcce3480f2645
I20240501 07:58:47.994204 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 0c86818cabaf49678cfbcce3480f2645 (table rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]) successfully deleted
I20240501 07:58:47.994438 12253 raft_consensus.cc:504] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.994743 12253 leader_election.cc:288] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:47.995093 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a3f9612e74665b0c8440918404350" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:47.995200 9271 raft_consensus.cc:3037] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.995191 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9a3f9612e74665b0c8440918404350" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:47.995278 9535 raft_consensus.cc:3037] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:47.996206 9535 raft_consensus.cc:2454] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:47.996585 9271 raft_consensus.cc:2454] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:47.996667 7077 leader_election.cc:302] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:47.996893 12253 raft_consensus.cc:2781] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:47.997066 12253 raft_consensus.cc:686] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:47.997217 12253 consensus_queue.cc:237] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:47.999378 6736 catalog_manager.cc:5484] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.050338 9535 consensus_queue.cc:237] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.051729 7008 raft_consensus.cc:1261] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.051792 9271 raft_consensus.cc:1261] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:48.052028 9469 consensus_peers.cc:485] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a81efefaf414a6091da247b0986c322. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.052165 12665 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.052345 11763 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.053994 12665 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.054436 9271 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.056104 7008 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.056700 6736 catalog_manager.cc:5484] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.077363 7140 consensus_queue.cc:237] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.078893 6876 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.079102 7008 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:48.079200 7077 consensus_peers.cc:485] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f723d182d04d439da6ab5b811448cf2d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.079437 12756 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.079631 12695 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.081666 12695 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:48.082064 7008 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:48.084323 6876 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:48.084965 6732 catalog_manager.cc:5484] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:48.106694 9535 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.107972 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:48.108147 9266 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.108251 9666 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.109416 11763 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.109601 12665 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:48.109967 9469 consensus_peers.cc:485] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8921a8351e04daba09a8e7e4f683fd9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.112438 9266 raft_consensus.cc:1261] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.112378 12665 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.112798 12253 consensus_queue.cc:1035] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.113283 9535 raft_consensus.cc:1261] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.113631 12253 consensus_queue.cc:1035] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.114840 9666 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.114998 9271 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.117085 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 9d9a3f9612e74665b0c8440918404350 (table rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d]). Expected version 1 got 0
I20240501 07:58:48.118136 6728 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.121666 12786 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.122084 12786 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:48.122692 12787 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.122854 6730 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 9d9a3f9612e74665b0c8440918404350 (table rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d]). Expected version 1 got 0
I20240501 07:58:48.123057 12787 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:48.124497 12785 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.124868 12785 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:48.126192 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d] alter complete (version 1)
I20240501 07:58:48.130271 11781 consensus_queue.cc:1035] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.133250 12608 consensus_queue.cc:1035] T bb0a970a4e624499aa3d51e1bc961cfd P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.147891 7140 consensus_queue.cc:237] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.149461 6876 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.149538 7008 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.150413 12756 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:48.150628 7076 consensus_peers.cc:485] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82c30f6292754d97a839700be93f8c07. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.150919 12756 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.152804 12756 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.153331 7008 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.155052 6876 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.155941 6730 catalog_manager.cc:5484] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.167538 9666 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:58:48.167948 11763 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:48.179865 7271 consensus_queue.cc:237] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.181213 9535 raft_consensus.cc:1261] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.181371 9401 raft_consensus.cc:1261] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.181739 11116 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:48.181949 7207 consensus_peers.cc:485] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22cb0ec763ff4941acbfbb007995241d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.181949 11820 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.183694 11820 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.184126 9401 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.185989 9535 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.186764 6736 catalog_manager.cc:5484] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.187448 9266 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:58:48.189489 11763 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:48.192075 7271 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:58:48.192850 12665 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:48.219769 9666 consensus_queue.cc:237] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:48.221329 9266 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.221532 9535 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.221915 12844 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.222119 12574 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:48.222290 9602 consensus_peers.cc:485] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f3e4cd27e4545baa255a01023cf1d68. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.224292 12574 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:48.226229 9535 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:48.226281 9266 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:48.228335 6728 catalog_manager.cc:5484] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:48.230791 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:48.236209 12785 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.236546 12785 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:48.236641 12787 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.236968 12787 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:48.237622 12786 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.238070 12786 tablet.cc:1620] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:48.243393 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e0d6282c4fe6462386ca71ebd0b3ce9d] alter complete (version 2)
I20240501 07:58:48.253582 12844 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.253978 7008 consensus_queue.cc:237] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.254745 12844 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:58:48.255841 6945 consensus_peers.cc:485] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2f299abad9e415283470b4b214fc62f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.256588 7271 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.257169 7140 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.257439 12648 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.257589 12172 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.259616 12688 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.260090 7271 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.261318 7140 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.263108 6732 catalog_manager.cc:5484] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:48.281989 7140 consensus_queue.cc:237] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.283578 7271 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:48.283821 7076 consensus_peers.cc:485] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2eec88312feb44feb70401fe29a3bb83. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.283833 6876 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:48.283963 12695 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.284296 12805 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:48.286164 12805 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.286594 6876 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.288595 7271 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.291697 6730 catalog_manager.cc:5484] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:48.301399 6876 raft_consensus.cc:3037] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.302866 12080 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.306092 12574 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.312461 9535 consensus_queue.cc:237] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:58:48.314082 9469 consensus_peers.cc:485] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1037ad1dda6448f7a026fc685642d5c6. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:48.314131 9666 raft_consensus.cc:1261] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.314128 9401 raft_consensus.cc:1261] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.314479 11487 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.314612 11763 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:48.317348 11763 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.318768 9666 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.321743 9401 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.322117 6732 catalog_manager.cc:5484] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:48.345402 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:48.351189 12785 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:48.351461 12786 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:48.355211 12787 tablet.cc:1722] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:48.358757 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=e0d6282c4fe6462386ca71ebd0b3ce9d] alter complete (version 3)
I20240501 07:58:48.389504 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:48.398928 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.399296 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.399345 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 0d818b9a4d9d48d9bab2b1c621db6828 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.399585 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d818b9a4d9d48d9bab2b1c621db6828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.402002 9251 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0f1d233434d729c88e2cbf3d92d21 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.402096 12564 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:48.402304 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0f1d233434d729c88e2cbf3d92d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.402973 12564 tablet_bootstrap.cc:654] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.403792 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.404067 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.404848 12565 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:48.405928 12950 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:48.406216 12565 tablet_bootstrap.cc:654] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.406731 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 0d818b9a4d9d48d9bab2b1c621db6828 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.407317 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.407577 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d818b9a4d9d48d9bab2b1c621db6828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.407639 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 0d818b9a4d9d48d9bab2b1c621db6828 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.407696 9382 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0f1d233434d729c88e2cbf3d92d21 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.407971 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d818b9a4d9d48d9bab2b1c621db6828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.406731 9644 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0f1d233434d729c88e2cbf3d92d21 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:48.408214 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0f1d233434d729c88e2cbf3d92d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.410470 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.410703 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0f1d233434d729c88e2cbf3d92d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.414959 12950 tablet_bootstrap.cc:654] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.415798 12952 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.416872 12952 tablet_bootstrap.cc:654] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.417243 12953 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:48.418121 12950 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:48.418282 12953 tablet_bootstrap.cc:654] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.418282 12950 ts_tablet_manager.cc:1367] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:48.418578 12565 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:48.418695 12565 ts_tablet_manager.cc:1367] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240501 07:58:48.418933 12950 raft_consensus.cc:348] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.419042 12564 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:48.419061 12950 raft_consensus.cc:374] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.419093 12950 raft_consensus.cc:729] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.419131 12564 ts_tablet_manager.cc:1367] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:58:48.419363 12950 consensus_queue.cc:260] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.421329 12564 raft_consensus.cc:348] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.421442 12564 raft_consensus.cc:374] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.421483 12564 raft_consensus.cc:729] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.421532 12565 raft_consensus.cc:348] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.421603 12564 consensus_queue.cc:260] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.421645 12565 raft_consensus.cc:374] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.421684 12565 raft_consensus.cc:729] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.421813 12950 ts_tablet_manager.cc:1398] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:48.421861 12564 ts_tablet_manager.cc:1398] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:48.421829 12565 consensus_queue.cc:260] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.422044 12564 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:48.422153 12565 ts_tablet_manager.cc:1398] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:48.423107 12564 tablet_bootstrap.cc:654] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.424235 12952 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:48.424448 12952 ts_tablet_manager.cc:1367] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:48.426563 12952 raft_consensus.cc:348] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.426827 12952 raft_consensus.cc:374] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.426910 12952 raft_consensus.cc:729] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.427101 12952 consensus_queue.cc:260] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.427654 12952 ts_tablet_manager.cc:1398] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:48.427841 12952 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.429291 12952 tablet_bootstrap.cc:654] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.429770 12564 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:48.429873 12564 ts_tablet_manager.cc:1367] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 07:58:48.430280 12564 raft_consensus.cc:348] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.430330 12953 tablet_bootstrap.cc:492] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:48.430382 12564 raft_consensus.cc:374] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.430419 12953 ts_tablet_manager.cc:1367] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:48.430419 12564 raft_consensus.cc:729] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.430531 12564 consensus_queue.cc:260] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.430769 12564 ts_tablet_manager.cc:1398] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:48.430819 12953 raft_consensus.cc:348] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.430950 12953 raft_consensus.cc:374] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.430980 12953 raft_consensus.cc:729] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.431161 12953 consensus_queue.cc:260] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.435683 12953 ts_tablet_manager.cc:1398] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:48.436311 12953 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:48.437202 12953 tablet_bootstrap.cc:654] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.440058 12952 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:48.440189 12952 ts_tablet_manager.cc:1367] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:48.440642 12952 raft_consensus.cc:348] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.440822 12952 raft_consensus.cc:374] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.440904 12952 raft_consensus.cc:729] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.440861 12953 tablet_bootstrap.cc:492] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:48.441102 12952 consensus_queue.cc:260] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.441157 12953 ts_tablet_manager.cc:1367] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:48.441601 12952 ts_tablet_manager.cc:1398] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:48.441699 12953 raft_consensus.cc:348] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.441776 12952 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.441813 12953 raft_consensus.cc:374] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.441849 12953 raft_consensus.cc:729] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.442015 12953 consensus_queue.cc:260] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.443249 12953 ts_tablet_manager.cc:1398] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:48.444558 12952 tablet_bootstrap.cc:654] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.445541 12952 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:48.445675 12952 ts_tablet_manager.cc:1367] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:48.446259 12952 raft_consensus.cc:348] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.446383 12952 raft_consensus.cc:374] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.446424 12952 raft_consensus.cc:729] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.446642 12952 consensus_queue.cc:260] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.447360 12952 ts_tablet_manager.cc:1398] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:48.459990 6729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:48.460139 6729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:48.460822 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:48.463227 6729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 9d9a3f9612e74665b0c8440918404350
I20240501 07:58:48.463559 9645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:48.463838 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a3f9612e74665b0c8440918404350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:48 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:48.463989 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a3f9612e74665b0c8440918404350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:48 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:48.464215 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9a3f9612e74665b0c8440918404350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:48 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:48.464509 12976 tablet_replica.cc:330] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:48.464669 12976 raft_consensus.cc:2229] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:48.464892 12976 raft_consensus.cc:2258] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:48.466132 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:48.466411 7252 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:48.466723 12976 ts_tablet_manager.cc:1875] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:48.467622 12975 tablet_replica.cc:330] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:48.467806 12975 raft_consensus.cc:2229] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:48.467948 12975 raft_consensus.cc:2258] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:48.468734 12975 ts_tablet_manager.cc:1875] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:48.470249 12974 tablet_replica.cc:330] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:48.470458 12974 raft_consensus.cc:2229] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:48.470650 12974 raft_consensus.cc:2258] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:48.471372 12974 ts_tablet_manager.cc:1875] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:48.471791 12975 ts_tablet_manager.cc:1888] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:48.472216 12975 log.cc:1196] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9d9a3f9612e74665b0c8440918404350
I20240501 07:58:48.472776 12975 ts_tablet_manager.cc:1909] T 9d9a3f9612e74665b0c8440918404350 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:48.474615 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 9d9a3f9612e74665b0c8440918404350 (table rebalancer_dynamic_table_new_name [id=e0d6282c4fe6462386ca71ebd0b3ce9d]) successfully deleted
I20240501 07:58:48.475436 12974 ts_tablet_manager.cc:1888] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:48.475601 12974 log.cc:1196] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9d9a3f9612e74665b0c8440918404350
I20240501 07:58:48.475999 12974 ts_tablet_manager.cc:1909] T 9d9a3f9612e74665b0c8440918404350 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:48.476898 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 9d9a3f9612e74665b0c8440918404350 (table rebalancer_dynamic_table_new_name [id=e0d6282c4fe6462386ca71ebd0b3ce9d]) successfully deleted
I20240501 07:58:48.486505 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:48.486891 9514 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:48.489784 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:48.495515 9382 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:48.527531 12976 ts_tablet_manager.cc:1888] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:48.527725 12976 log.cc:1196] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d9a3f9612e74665b0c8440918404350
I20240501 07:58:48.528589 12976 ts_tablet_manager.cc:1909] T 9d9a3f9612e74665b0c8440918404350 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:48.530273 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 9d9a3f9612e74665b0c8440918404350 (table rebalancer_dynamic_table_new_name [id=e0d6282c4fe6462386ca71ebd0b3ce9d]) successfully deleted
I20240501 07:58:48.565357 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:48.571774 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 67669b63292a4028b12158049d16c48c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]), partition=
I20240501 07:58:48.571799 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 67669b63292a4028b12158049d16c48c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]), partition=
I20240501 07:58:48.572065 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67669b63292a4028b12158049d16c48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.572067 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67669b63292a4028b12158049d16c48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.574018 12952 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.574108 12565 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:48.574862 12952 tablet_bootstrap.cc:654] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.574939 12565 tablet_bootstrap.cc:654] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.575775 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 67669b63292a4028b12158049d16c48c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]), partition=
I20240501 07:58:48.575974 12952 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:48.576030 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67669b63292a4028b12158049d16c48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.576063 12952 ts_tablet_manager.cc:1367] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:48.576452 12952 raft_consensus.cc:348] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.576543 12952 raft_consensus.cc:374] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.576577 12952 raft_consensus.cc:729] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.576619 12565 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:48.576710 12565 ts_tablet_manager.cc:1367] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:48.576697 12952 consensus_queue.cc:260] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.576936 12952 ts_tablet_manager.cc:1398] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:48.577152 12565 raft_consensus.cc:348] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.577247 12565 raft_consensus.cc:374] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.577280 12565 raft_consensus.cc:729] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.577397 12565 consensus_queue.cc:260] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.577637 12565 ts_tablet_manager.cc:1398] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:48.582747 12990 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:48.583724 12990 tablet_bootstrap.cc:654] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:48.584721 12990 tablet_bootstrap.cc:492] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:48.584821 12990 ts_tablet_manager.cc:1367] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:48.585227 12990 raft_consensus.cc:348] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.585330 12990 raft_consensus.cc:374] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:48.585361 12990 raft_consensus.cc:729] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.585494 12990 consensus_queue.cc:260] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.585762 12990 ts_tablet_manager.cc:1398] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:48.596354 12997 ts_tablet_manager.cc:902] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:48.597194 12997 tablet_copy_client.cc:308] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:58:48.597924 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet f8921a8351e04daba09a8e7e4f683fd9 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:43037)
I20240501 07:58:48.598063 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet f8921a8351e04daba09a8e7e4f683fd9 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:43037: session id = fc946354b5d04503a56122e59fc320be-f8921a8351e04daba09a8e7e4f683fd9
I20240501 07:58:48.599408 9555 tablet_copy_source_session.cc:205] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.600015 11487 raft_consensus.cc:482] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:48.600090 11487 raft_consensus.cc:504] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.600201 11060 raft_consensus.cc:482] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:48.600301 11060 raft_consensus.cc:504] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.600421 11487 leader_election.cc:288] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:48.600704 12997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8921a8351e04daba09a8e7e4f683fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.600768 11060 leader_election.cc:288] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:48.600893 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:48.601015 9666 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:48.601603 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:48.601688 9470 leader_election.cc:302] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:48.601728 6876 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:48.601979 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0f1d233434d729c88e2cbf3d92d21" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:48.602133 9266 raft_consensus.cc:2454] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:48.602412 9338 leader_election.cc:302] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:48.602572 11060 raft_consensus.cc:2781] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:48.602625 11060 raft_consensus.cc:482] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:48.602667 11060 raft_consensus.cc:3037] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.602638 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0f1d233434d729c88e2cbf3d92d21" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:48.602771 9666 raft_consensus.cc:2454] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:48.603091 11487 raft_consensus.cc:2781] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:48.603168 11487 raft_consensus.cc:482] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:48.603211 11487 raft_consensus.cc:3037] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.603451 12997 tablet_copy_client.cc:783] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.603646 12997 tablet_copy_client.cc:647] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.603891 11060 raft_consensus.cc:504] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.604228 11060 leader_election.cc:288] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:48.604339 11487 raft_consensus.cc:504] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.604656 11487 leader_election.cc:288] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:48.604955 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:48.605054 9666 raft_consensus.cc:3037] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.605363 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0f1d233434d729c88e2cbf3d92d21" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:48.605425 12997 tablet_copy_client.cc:523] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.605474 9266 raft_consensus.cc:3037] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.605832 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:48.605935 6876 raft_consensus.cc:3037] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.606007 9666 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:48.606379 9266 raft_consensus.cc:2454] T a3b0f1d233434d729c88e2cbf3d92d21 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:48.606892 6876 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:48.607074 12997 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:48.607080 9470 leader_election.cc:302] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:48.607162 9338 leader_election.cc:302] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:48.607347 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0f1d233434d729c88e2cbf3d92d21" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:48.607436 9666 raft_consensus.cc:3037] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.608175 9666 raft_consensus.cc:2454] T a3b0f1d233434d729c88e2cbf3d92d21 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:48.608665 11487 raft_consensus.cc:2781] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:48.608765 11487 raft_consensus.cc:686] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:48.608899 11487 consensus_queue.cc:237] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:48.611464 6736 catalog_manager.cc:5484] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:48.612816 11060 raft_consensus.cc:2781] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:48.612910 11060 raft_consensus.cc:686] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:48.613051 11060 consensus_queue.cc:237] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:48.620654 12997 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.620635 6728 catalog_manager.cc:5484] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:48.620941 12997 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:58:48.621127 12997 ts_tablet_manager.cc:1367] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 07:58:48.621541 12997 raft_consensus.cc:348] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.621636 12997 raft_consensus.cc:729] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.621791 12997 consensus_queue.cc:260] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.622136 12997 ts_tablet_manager.cc:1398] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:48.623056 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-f8921a8351e04daba09a8e7e4f683fd9 received from {username='slave'} at 127.0.99.131:43037
I20240501 07:58:48.623152 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fc946354b5d04503a56122e59fc320be-f8921a8351e04daba09a8e7e4f683fd9 on tablet f8921a8351e04daba09a8e7e4f683fd9 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:58:48.625519 12080 raft_consensus.cc:482] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:48.625641 12080 raft_consensus.cc:504] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.626148 12080 leader_election.cc:288] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:48.626591 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67669b63292a4028b12158049d16c48c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:48.626713 7271 raft_consensus.cc:2454] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:48.627066 9600 leader_election.cc:302] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:48.627285 12080 raft_consensus.cc:2781] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:48.627341 12080 raft_consensus.cc:482] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:48.627384 12080 raft_consensus.cc:3037] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.627926 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67669b63292a4028b12158049d16c48c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:48.628067 9266 raft_consensus.cc:2454] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:48.628419 12080 raft_consensus.cc:504] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.628765 12080 leader_election.cc:288] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:48.629148 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67669b63292a4028b12158049d16c48c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:48.629259 7271 raft_consensus.cc:3037] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.630278 7271 raft_consensus.cc:2454] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:48.630550 9600 leader_election.cc:302] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:48.631210 12080 raft_consensus.cc:2781] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:48.631302 12080 raft_consensus.cc:686] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:48.631422 12080 consensus_queue.cc:237] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:48.634527 6737 catalog_manager.cc:5484] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:48.636245 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67669b63292a4028b12158049d16c48c" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:48.636365 9266 raft_consensus.cc:3037] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.637077 9266 raft_consensus.cc:2454] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:48.641000 12997 ts_tablet_manager.cc:902] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:48.641487 12997 tablet_copy_client.cc:308] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:58:48.643263 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 0a81efefaf414a6091da247b0986c322 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:43037)
I20240501 07:58:48.643383 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 0a81efefaf414a6091da247b0986c322 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:43037: session id = fc946354b5d04503a56122e59fc320be-0a81efefaf414a6091da247b0986c322
I20240501 07:58:48.644281 9555 tablet_copy_source_session.cc:205] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.645053 12997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a81efefaf414a6091da247b0986c322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.647444 12997 tablet_copy_client.cc:783] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.647612 12997 tablet_copy_client.cc:647] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.650522 12997 tablet_copy_client.cc:523] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.651998 12997 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:48.656270 12997 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.656518 12997 tablet_bootstrap.cc:492] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:58:48.656685 12997 ts_tablet_manager.cc:1367] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:48.657213 12997 raft_consensus.cc:348] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.657339 12997 raft_consensus.cc:729] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.657464 12997 consensus_queue.cc:260] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.657760 12997 ts_tablet_manager.cc:1398] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:48.658252 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-0a81efefaf414a6091da247b0986c322 received from {username='slave'} at 127.0.99.131:43037
I20240501 07:58:48.658332 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fc946354b5d04503a56122e59fc320be-0a81efefaf414a6091da247b0986c322 on tablet 0a81efefaf414a6091da247b0986c322 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:58:48.675174 13036 ts_tablet_manager.cc:902] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:48.675647 13036 tablet_copy_client.cc:308] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:48.676103 7161 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet f723d182d04d439da6ab5b811448cf2d from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:53675)
I20240501 07:58:48.676211 7161 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet f723d182d04d439da6ab5b811448cf2d from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:53675: session id = ac6c8bc1b74b483791e262abee7862cf-f723d182d04d439da6ab5b811448cf2d
I20240501 07:58:48.676914 7161 tablet_copy_source_session.cc:205] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.677659 13036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f723d182d04d439da6ab5b811448cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.680194 13041 ts_tablet_manager.cc:902] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:48.680240 13036 tablet_copy_client.cc:783] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.680413 13036 tablet_copy_client.cc:647] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.680732 13041 tablet_copy_client.cc:308] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:48.681166 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 2eec88312feb44feb70401fe29a3bb83 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:58:48.681269 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 2eec88312feb44feb70401fe29a3bb83 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-2eec88312feb44feb70401fe29a3bb83
I20240501 07:58:48.681610 13036 tablet_copy_client.cc:523] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.682039 7160 tablet_copy_source_session.cc:205] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.682811 13036 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:48.683351 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eec88312feb44feb70401fe29a3bb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.685686 13041 tablet_copy_client.cc:783] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.685855 13041 tablet_copy_client.cc:647] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.686851 13041 tablet_copy_client.cc:523] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.688098 13041 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:48.710749 13036 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.711149 13036 tablet_bootstrap.cc:492] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:48.711338 13036 ts_tablet_manager.cc:1367] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.005s
I20240501 07:58:48.711807 13036 raft_consensus.cc:348] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.711980 13036 raft_consensus.cc:729] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.712109 13036 consensus_queue.cc:260] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.712508 13036 ts_tablet_manager.cc:1398] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:48.713712 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-f723d182d04d439da6ab5b811448cf2d received from {username='slave'} at 127.0.99.135:53675
I20240501 07:58:48.713793 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-f723d182d04d439da6ab5b811448cf2d on tablet f723d182d04d439da6ab5b811448cf2d with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:48.714447 13041 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.714738 13048 ts_tablet_manager.cc:902] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:48.714751 13041 tablet_bootstrap.cc:492] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:48.714932 13041 ts_tablet_manager.cc:1367] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.002s
I20240501 07:58:48.715332 13048 tablet_copy_client.cc:308] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:58:48.715438 13041 raft_consensus.cc:348] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.715554 13041 raft_consensus.cc:729] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.715710 13041 consensus_queue.cc:260] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.716027 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 22cb0ec763ff4941acbfbb007995241d from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:49987)
I20240501 07:58:48.716126 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 22cb0ec763ff4941acbfbb007995241d from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:49987: session id = 49adb9e99822494196ed22044bfed81f-22cb0ec763ff4941acbfbb007995241d
I20240501 07:58:48.716612 13041 ts_tablet_manager.cc:1398] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:48.717008 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-2eec88312feb44feb70401fe29a3bb83 received from {username='slave'} at 127.0.99.133:44859
I20240501 07:58:48.717078 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-2eec88312feb44feb70401fe29a3bb83 on tablet 2eec88312feb44feb70401fe29a3bb83 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:48.717077 7292 tablet_copy_source_session.cc:205] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.718334 13048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cb0ec763ff4941acbfbb007995241d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.721194 13048 tablet_copy_client.cc:783] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.721356 13048 tablet_copy_client.cc:647] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.721606 13053 ts_tablet_manager.cc:902] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:48.722364 13053 tablet_copy_client.cc:308] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:58:48.722461 13048 tablet_copy_client.cc:523] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.723110 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 4f3e4cd27e4545baa255a01023cf1d68 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:59385)
I20240501 07:58:48.723253 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 4f3e4cd27e4545baa255a01023cf1d68 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:59385: session id = 9a3888360d4d4ace96b6b909c160bfc2-4f3e4cd27e4545baa255a01023cf1d68
I20240501 07:58:48.723842 13048 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:48.724555 9686 tablet_copy_source_session.cc:205] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.726029 13053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3e4cd27e4545baa255a01023cf1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.727530 13048 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.727762 13048 tablet_bootstrap.cc:492] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:48.727911 13048 ts_tablet_manager.cc:1367] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 07:58:48.728374 13048 raft_consensus.cc:348] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.728489 13048 raft_consensus.cc:729] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.728610 13048 consensus_queue.cc:260] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:48.728705 13053 tablet_copy_client.cc:783] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.728902 13048 ts_tablet_manager.cc:1398] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:48.728972 13053 tablet_copy_client.cc:647] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.730206 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-22cb0ec763ff4941acbfbb007995241d received from {username='slave'} at 127.0.99.133:49987
I20240501 07:58:48.730279 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session 49adb9e99822494196ed22044bfed81f-22cb0ec763ff4941acbfbb007995241d on tablet 22cb0ec763ff4941acbfbb007995241d with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:48.730978 13053 tablet_copy_client.cc:523] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.732606 13053 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:48.735221 13059 ts_tablet_manager.cc:902] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:48.735882 13059 tablet_copy_client.cc:308] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:48.736621 13053 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.736905 13053 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:58:48.737071 13053 ts_tablet_manager.cc:1367] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:48.737665 13053 raft_consensus.cc:348] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:48.737766 13053 raft_consensus.cc:729] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.737908 13053 consensus_queue.cc:260] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:48.738188 13053 ts_tablet_manager.cc:1398] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:48.738700 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-4f3e4cd27e4545baa255a01023cf1d68 received from {username='slave'} at 127.0.99.130:59385
I20240501 07:58:48.738786 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-4f3e4cd27e4545baa255a01023cf1d68 on tablet 4f3e4cd27e4545baa255a01023cf1d68 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:58:48.743393 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 82c30f6292754d97a839700be93f8c07 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:38089)
I20240501 07:58:48.743496 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 82c30f6292754d97a839700be93f8c07 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:38089: session id = 7fa3a88c25964445975bb47e0feed460-82c30f6292754d97a839700be93f8c07
I20240501 07:58:48.744248 7160 tablet_copy_source_session.cc:205] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.745613 13059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30f6292754d97a839700be93f8c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.747334 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:48.747751 13059 tablet_copy_client.cc:783] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.747911 13059 tablet_copy_client.cc:647] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.749114 13059 tablet_copy_client.cc:523] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.750312 13059 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.751350 7271 raft_consensus.cc:1261] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.751591 9271 raft_consensus.cc:1261] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:48.751817 12080 consensus_queue.cc:1035] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.752750 12574 consensus_queue.cc:1035] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:48.754333 6734 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]). Expected version 1 got 0
I20240501 07:58:48.756161 6734 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]). Expected version 1 got 0
I20240501 07:58:48.757087 12786 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.756834 13070 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.757419 13070 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:48.757429 12786 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:48.763661 13071 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:48.763067 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f] alter complete (version 1)
I20240501 07:58:48.764564 6733 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f]). Expected version 1 got 0
I20240501 07:58:48.765216 13071 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:48.768644 11487 raft_consensus.cc:482] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:48.768745 11487 raft_consensus.cc:504] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.769179 11487 leader_election.cc:288] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:48.769568 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:48.769629 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:48.769683 9665 raft_consensus.cc:2454] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:48.769769 9401 raft_consensus.cc:2454] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:48.770041 9470 leader_election.cc:302] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:48.770280 11487 raft_consensus.cc:2781] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:48.770326 11487 raft_consensus.cc:482] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:48.770367 11487 raft_consensus.cc:3037] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.771198 11487 raft_consensus.cc:504] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.771518 11487 leader_election.cc:288] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:48.771903 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:48.772015 9665 raft_consensus.cc:3037] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.772080 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:48.772183 9401 raft_consensus.cc:3037] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:48.772838 9665 raft_consensus.cc:2454] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:48.773131 9470 leader_election.cc:302] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:48.773164 9401 raft_consensus.cc:2454] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:48.773285 11487 raft_consensus.cc:2781] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:48.773361 11487 raft_consensus.cc:686] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:48.773490 11487 consensus_queue.cc:237] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:48.775413 6733 catalog_manager.cc:5484] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:48.782413 13059 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.782689 13059 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:48.782874 13059 ts_tablet_manager.cc:1367] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.000s
I20240501 07:58:48.783301 13059 raft_consensus.cc:348] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.783409 13059 raft_consensus.cc:729] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.783528 13059 consensus_queue.cc:260] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.783769 13059 ts_tablet_manager.cc:1398] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:48.784242 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-82c30f6292754d97a839700be93f8c07 received from {username='slave'} at 127.0.99.136:38089
I20240501 07:58:48.784334 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-82c30f6292754d97a839700be93f8c07 on tablet 82c30f6292754d97a839700be93f8c07 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:48.806973 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:48.809217 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:48.814563 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:48.815925 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:48.823170 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:48.831507 7252 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:48.832266 9382 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:48.846954 9645 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:48.861182 13059 ts_tablet_manager.cc:902] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:48.863905 13059 tablet_copy_client.cc:308] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:48.864384 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet c2f299abad9e415283470b4b214fc62f from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:39427)
I20240501 07:58:48.864506 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet c2f299abad9e415283470b4b214fc62f from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:39427: session id = 7fa3a88c25964445975bb47e0feed460-c2f299abad9e415283470b4b214fc62f
I20240501 07:58:48.865320 7030 tablet_copy_source_session.cc:205] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.866237 13059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f299abad9e415283470b4b214fc62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.867501 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:48.868594 13059 tablet_copy_client.cc:783] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.868741 13059 tablet_copy_client.cc:647] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.871205 13059 tablet_copy_client.cc:523] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.873569 13070 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.873854 13059 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:48.873878 13070 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:48.874037 13071 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.874312 13071 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:48.874691 12786 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:48.874959 12786 tablet.cc:1620] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:48.880502 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=71a379c514834d5f93ed62272f1e298f] alter complete (version 2)
I20240501 07:58:48.899405 13059 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.899694 13059 tablet_bootstrap.cc:492] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:48.899890 13059 ts_tablet_manager.cc:1367] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 07:58:48.900470 13059 raft_consensus.cc:348] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.900600 13059 raft_consensus.cc:729] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.900740 13059 consensus_queue.cc:260] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:48.901536 13059 ts_tablet_manager.cc:1398] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:48.902915 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-c2f299abad9e415283470b4b214fc62f received from {username='slave'} at 127.0.99.136:39427
I20240501 07:58:48.902990 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-c2f299abad9e415283470b4b214fc62f on tablet c2f299abad9e415283470b4b214fc62f with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:48.938962 12997 ts_tablet_manager.cc:902] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:48.939432 12997 tablet_copy_client.cc:308] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:58:48.939842 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 1037ad1dda6448f7a026fc685642d5c6 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:43037)
I20240501 07:58:48.939949 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 1037ad1dda6448f7a026fc685642d5c6 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:43037: session id = fc946354b5d04503a56122e59fc320be-1037ad1dda6448f7a026fc685642d5c6
I20240501 07:58:48.941031 9555 tablet_copy_source_session.cc:205] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.942023 12997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037ad1dda6448f7a026fc685642d5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.944315 12997 tablet_copy_client.cc:783] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.944461 12997 tablet_copy_client.cc:647] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.946354 12997 tablet_copy_client.cc:523] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:48.947623 12997 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:48.951808 12997 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:48.952062 12997 tablet_bootstrap.cc:492] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:58:48.952239 12997 ts_tablet_manager.cc:1367] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:48.952733 12997 raft_consensus.cc:348] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.952865 12997 raft_consensus.cc:729] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:48.953006 12997 consensus_queue.cc:260] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:48.953315 12997 ts_tablet_manager.cc:1398] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:48.953950 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-1037ad1dda6448f7a026fc685642d5c6 received from {username='slave'} at 127.0.99.131:43037
I20240501 07:58:48.954041 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fc946354b5d04503a56122e59fc320be-1037ad1dda6448f7a026fc685642d5c6 on tablet 1037ad1dda6448f7a026fc685642d5c6 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:58:48.984367 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:48.986414 7252 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:48.991675 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:48.994516 9645 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:48.997329 13070 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:48.999339 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:48.999441 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:48.999589 12786 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:49.003095 11060 consensus_queue.cc:1035] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.014214 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=71a379c514834d5f93ed62272f1e298f] alter complete (version 3)
I20240501 07:58:49.015195 13071 tablet.cc:1722] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:49.015682 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.022799 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:49.025890 12665 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.028658 9382 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:49.050225 12665 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.052579 12912 consensus_queue.cc:1035] T a3b0f1d233434d729c88e2cbf3d92d21 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.076177 9271 raft_consensus.cc:1203] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:49.080235 7140 raft_consensus.cc:1203] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:49.120275 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:49.120427 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:49.123083 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 67669b63292a4028b12158049d16c48c
I20240501 07:58:49.124128 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 67669b63292a4028b12158049d16c48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:49.124461 13177 tablet_replica.cc:330] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:49.124583 13177 raft_consensus.cc:2229] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.124589 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 67669b63292a4028b12158049d16c48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:49.124676 13177 raft_consensus.cc:2258] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.124925 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet 67669b63292a4028b12158049d16c48c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:49.125083 13178 tablet_replica.cc:330] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:49.125224 13179 tablet_replica.cc:330] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:49.125232 13178 raft_consensus.cc:2229] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:49.125301 13177 ts_tablet_manager.cc:1875] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.125413 13179 raft_consensus.cc:2229] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.125552 13179 raft_consensus.cc:2258] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.125512 13178 raft_consensus.cc:2258] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.126188 13179 ts_tablet_manager.cc:1875] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.126222 13178 ts_tablet_manager.cc:1875] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.127782 13177 ts_tablet_manager.cc:1888] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.127926 13177 log.cc:1196] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/67669b63292a4028b12158049d16c48c
I20240501 07:58:49.128417 13177 ts_tablet_manager.cc:1909] T 67669b63292a4028b12158049d16c48c P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:49.128551 13179 ts_tablet_manager.cc:1888] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.128674 13179 log.cc:1196] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/67669b63292a4028b12158049d16c48c
I20240501 07:58:49.128892 13178 ts_tablet_manager.cc:1888] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.128978 13178 log.cc:1196] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/67669b63292a4028b12158049d16c48c
I20240501 07:58:49.129133 13179 ts_tablet_manager.cc:1909] T 67669b63292a4028b12158049d16c48c P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:49.129266 13178 ts_tablet_manager.cc:1909] T 67669b63292a4028b12158049d16c48c P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:49.129356 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table_new_name [id=71a379c514834d5f93ed62272f1e298f]) successfully deleted
I20240501 07:58:49.129961 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table_new_name [id=71a379c514834d5f93ed62272f1e298f]) successfully deleted
I20240501 07:58:49.130546 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 67669b63292a4028b12158049d16c48c (table rebalancer_dynamic_table_new_name [id=71a379c514834d5f93ed62272f1e298f]) successfully deleted
I20240501 07:58:49.135668 7140 raft_consensus.cc:1203] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:49.182230 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:49.182873 7008 raft_consensus.cc:1203] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:49.184424 9271 raft_consensus.cc:1203] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:49.185119 9645 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:49.187748 12665 consensus_queue.cc:1035] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.191444 11487 consensus_queue.cc:1035] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.200560 7252 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:49.202658 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:49.211189 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.216784 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:49.217118 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:49.219991 9382 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:49.231329 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:49.239761 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 774f9074f2b24c78bed9a4d705d2532f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]), partition=
I20240501 07:58:49.239871 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 774f9074f2b24c78bed9a4d705d2532f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]), partition=
I20240501 07:58:49.240073 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774f9074f2b24c78bed9a4d705d2532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.240128 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774f9074f2b24c78bed9a4d705d2532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.242277 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 774f9074f2b24c78bed9a4d705d2532f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]), partition=
I20240501 07:58:49.242571 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774f9074f2b24c78bed9a4d705d2532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.251039 13212 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:49.251382 13214 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:49.253445 13214 tablet_bootstrap.cc:654] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.254103 13212 tablet_bootstrap.cc:654] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.262022 13213 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:49.262423 9535 raft_consensus.cc:1203] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:49.265627 13213 tablet_bootstrap.cc:654] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.267364 13214 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:49.267511 13214 ts_tablet_manager.cc:1367] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:58:49.267961 13214 raft_consensus.cc:348] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.268097 13214 raft_consensus.cc:374] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.268582 13212 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:49.268705 13212 ts_tablet_manager.cc:1367] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:58:49.268762 13213 tablet_bootstrap.cc:492] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:49.268882 13213 ts_tablet_manager.cc:1367] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:49.269274 13212 raft_consensus.cc:348] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.269507 13212 raft_consensus.cc:374] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.269594 13212 raft_consensus.cc:729] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.269564 13213 raft_consensus.cc:348] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.269697 13214 raft_consensus.cc:729] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.269760 13213 raft_consensus.cc:374] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.269814 13213 raft_consensus.cc:729] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.270468 13213 consensus_queue.cc:260] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.270390 13212 consensus_queue.cc:260] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.269989 13214 consensus_queue.cc:260] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.270828 13213 ts_tablet_manager.cc:1398] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:49.271150 13212 ts_tablet_manager.cc:1398] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:49.272184 13214 ts_tablet_manager.cc:1398] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:49.291168 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:49.291337 9666 consensus_queue.cc:237] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:49.292872 7008 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.293921 9271 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.294512 12080 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.294828 12080 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.297711 12574 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:49.298298 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 9324ffeaa4124027b586425e86bc6de4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.298609 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324ffeaa4124027b586425e86bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.299908 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 9324ffeaa4124027b586425e86bc6de4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.300165 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324ffeaa4124027b586425e86bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.300880 13212 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:49.301815 13212 tablet_bootstrap.cc:654] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.302088 6736 catalog_manager.cc:5484] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:49.302609 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba12580263f4bd883ad7e003ebd6c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.302882 6858 tablet_service.cc:1467] Processing CreateTablet for tablet c0cf86e678084bc3b948968e5af0365d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.302913 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba12580263f4bd883ad7e003ebd6c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.303155 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0cf86e678084bc3b948968e5af0365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.303774 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba12580263f4bd883ad7e003ebd6c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.304014 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba12580263f4bd883ad7e003ebd6c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.303985 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 9324ffeaa4124027b586425e86bc6de4 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.304225 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9324ffeaa4124027b586425e86bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.305624 9646 tablet_service.cc:1467] Processing CreateTablet for tablet c0cf86e678084bc3b948968e5af0365d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.305824 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0cf86e678084bc3b948968e5af0365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.306476 9271 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:49.306782 7008 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:49.302837 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba12580263f4bd883ad7e003ebd6c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.308252 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba12580263f4bd883ad7e003ebd6c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.308966 13238 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:49.310002 13238 tablet_bootstrap.cc:654] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.310362 9382 tablet_service.cc:1467] Processing CreateTablet for tablet c0cf86e678084bc3b948968e5af0365d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:49.310601 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0cf86e678084bc3b948968e5af0365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.311300 13238 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:49.311422 13238 ts_tablet_manager.cc:1367] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:49.312076 13238 raft_consensus.cc:348] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.312222 13238 raft_consensus.cc:374] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.312275 13238 raft_consensus.cc:729] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.312331 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3e4cd27e4545baa255a01023cf1d68 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:49.312464 13238 consensus_queue.cc:260] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.312770 13238 ts_tablet_manager.cc:1398] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.313136 13239 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:49.313697 13242 tablet_replica.cc:330] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:49.313879 13242 raft_consensus.cc:2229] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.314177 13239 tablet_bootstrap.cc:654] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.314188 13242 raft_consensus.cc:2258] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.314630 13242 ts_tablet_manager.cc:1875] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:49.315358 13240 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:49.316689 13240 tablet_bootstrap.cc:654] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.318184 13236 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:49.318455 13242 ts_tablet_manager.cc:1888] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:49.318558 13242 log.cc:1196] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4f3e4cd27e4545baa255a01023cf1d68
I20240501 07:58:49.318598 13240 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:49.318708 13240 ts_tablet_manager.cc:1367] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:49.319196 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 4f3e4cd27e4545baa255a01023cf1d68 (table rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]) successfully deleted
I20240501 07:58:49.319464 13236 tablet_bootstrap.cc:654] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.319463 13240 raft_consensus.cc:348] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.319626 13240 raft_consensus.cc:374] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.319710 13240 raft_consensus.cc:729] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.319913 13212 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:49.320009 13212 ts_tablet_manager.cc:1367] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:58:49.320194 13240 consensus_queue.cc:260] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.320572 13239 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:49.320571 13240 ts_tablet_manager.cc:1398] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:49.320531 13212 raft_consensus.cc:348] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.320636 13212 raft_consensus.cc:374] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.320654 13239 ts_tablet_manager.cc:1367] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:49.320672 13212 raft_consensus.cc:729] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.320792 13240 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:49.320791 13212 consensus_queue.cc:260] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.321084 13212 ts_tablet_manager.cc:1398] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.321269 13212 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:49.321527 13236 tablet_bootstrap.cc:492] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:49.321605 13236 ts_tablet_manager.cc:1367] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:49.321954 13240 tablet_bootstrap.cc:654] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.322065 13236 raft_consensus.cc:348] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.322171 13236 raft_consensus.cc:374] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.322201 13236 raft_consensus.cc:729] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.322362 13212 tablet_bootstrap.cc:654] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.322326 13236 consensus_queue.cc:260] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.322412 13239 raft_consensus.cc:348] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.322518 13239 raft_consensus.cc:374] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.322553 13239 raft_consensus.cc:729] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.322644 13236 ts_tablet_manager.cc:1398] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.322712 13239 consensus_queue.cc:260] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.322929 13240 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:49.322999 13240 ts_tablet_manager.cc:1367] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:49.323102 13239 ts_tablet_manager.cc:1398] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:49.323336 13239 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:49.323427 13240 raft_consensus.cc:348] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.323522 13240 raft_consensus.cc:374] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.323550 13240 raft_consensus.cc:729] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.323655 13240 consensus_queue.cc:260] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.323963 13240 ts_tablet_manager.cc:1398] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.324124 13240 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:49.324973 13240 tablet_bootstrap.cc:654] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.326454 13239 tablet_bootstrap.cc:654] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.326560 13240 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:49.326646 13240 ts_tablet_manager.cc:1367] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:49.327677 9666 raft_consensus.cc:1203] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:49.335203 13240 raft_consensus.cc:348] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.335333 13240 raft_consensus.cc:374] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.335372 13240 raft_consensus.cc:729] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.335486 13240 consensus_queue.cc:260] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.336301 13240 ts_tablet_manager.cc:1398] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:58:49.337096 13212 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:49.337173 13212 ts_tablet_manager.cc:1367] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:58:49.337590 13239 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:49.337674 13239 ts_tablet_manager.cc:1367] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 07:58:49.337617 13212 raft_consensus.cc:348] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.337713 13212 raft_consensus.cc:374] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.337757 13212 raft_consensus.cc:729] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.337893 13212 consensus_queue.cc:260] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.338136 13212 ts_tablet_manager.cc:1398] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.338162 13239 raft_consensus.cc:348] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.338269 13239 raft_consensus.cc:374] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.338299 13239 raft_consensus.cc:729] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.338411 13239 consensus_queue.cc:260] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.340224 13239 ts_tablet_manager.cc:1398] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:49.364526 11060 raft_consensus.cc:482] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:49.364624 11060 raft_consensus.cc:504] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.365023 11060 leader_election.cc:288] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:49.365357 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0cf86e678084bc3b948968e5af0365d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:49.365456 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0cf86e678084bc3b948968e5af0365d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:49.365506 9666 raft_consensus.cc:2454] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:49.365597 6876 raft_consensus.cc:2454] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:49.366079 9338 leader_election.cc:302] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.366436 11060 raft_consensus.cc:2781] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:49.366493 11060 raft_consensus.cc:482] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:49.366529 11060 raft_consensus.cc:3037] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.367650 11060 raft_consensus.cc:504] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.368017 11060 leader_election.cc:288] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:49.368299 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0cf86e678084bc3b948968e5af0365d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:49.368412 9666 raft_consensus.cc:3037] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.368441 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0cf86e678084bc3b948968e5af0365d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:49.368564 6876 raft_consensus.cc:3037] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.369361 9666 raft_consensus.cc:2454] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:49.369419 6876 raft_consensus.cc:2454] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:49.369632 9338 leader_election.cc:302] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.369822 11060 raft_consensus.cc:2781] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:49.369915 11060 raft_consensus.cc:686] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:49.370025 11060 consensus_queue.cc:237] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:49.372243 6736 catalog_manager.cc:5484] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:49.374591 9666 raft_consensus.cc:1203] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:49.404006 7140 raft_consensus.cc:1203] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:49.434616 11060 raft_consensus.cc:482] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:49.434741 11060 raft_consensus.cc:504] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.435227 11060 leader_election.cc:288] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:49.435597 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774f9074f2b24c78bed9a4d705d2532f" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:49.435662 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774f9074f2b24c78bed9a4d705d2532f" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:49.435751 9271 raft_consensus.cc:2454] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:49.435788 9535 raft_consensus.cc:2454] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:49.436046 9338 leader_election.cc:302] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.436208 11060 raft_consensus.cc:2781] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:49.436262 11060 raft_consensus.cc:482] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:49.436308 11060 raft_consensus.cc:3037] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.437109 11060 raft_consensus.cc:504] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.437434 11060 leader_election.cc:288] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:49.437747 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774f9074f2b24c78bed9a4d705d2532f" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:49.437865 9271 raft_consensus.cc:3037] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.437811 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774f9074f2b24c78bed9a4d705d2532f" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:49.437971 9535 raft_consensus.cc:3037] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.438628 9271 raft_consensus.cc:2454] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:49.438742 9535 raft_consensus.cc:2454] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:49.438896 9338 leader_election.cc:302] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.439083 11060 raft_consensus.cc:2781] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:49.439198 11060 raft_consensus.cc:686] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:49.439347 11060 consensus_queue.cc:237] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:49.441309 6730 catalog_manager.cc:5484] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:49.455762 12648 raft_consensus.cc:482] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:49.455895 12648 raft_consensus.cc:504] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.456198 12648 leader_election.cc:288] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:49.456521 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324ffeaa4124027b586425e86bc6de4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:49.456650 9401 raft_consensus.cc:2454] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:49.456631 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324ffeaa4124027b586425e86bc6de4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:49.456781 9666 raft_consensus.cc:2454] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:49.456899 6945 leader_election.cc:302] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.457068 12648 raft_consensus.cc:2781] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:49.457134 12648 raft_consensus.cc:482] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:49.457178 12648 raft_consensus.cc:3037] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.458026 12648 raft_consensus.cc:504] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.458364 12648 leader_election.cc:288] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:49.458680 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324ffeaa4124027b586425e86bc6de4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:49.458680 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9324ffeaa4124027b586425e86bc6de4" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:49.458791 9666 raft_consensus.cc:3037] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.458814 9401 raft_consensus.cc:3037] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.459745 9666 raft_consensus.cc:2454] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:49.459745 9401 raft_consensus.cc:2454] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:49.459995 6945 leader_election.cc:302] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:49.460145 12648 raft_consensus.cc:2781] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:49.460289 12648 raft_consensus.cc:686] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:49.460414 12648 consensus_queue.cc:237] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:49.462122 6730 catalog_manager.cc:5484] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:49.544598 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:49.548525 9535 raft_consensus.cc:1261] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:49.548535 9271 raft_consensus.cc:1261] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:49.548959 12912 consensus_queue.cc:1035] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.549556 12912 consensus_queue.cc:1035] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.553786 13299 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:49.554054 6737 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]). Expected version 1 got 0
I20240501 07:58:49.554208 13298 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:49.554370 13299 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:49.554688 13298 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:49.557690 6730 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]). Expected version 1 got 0
I20240501 07:58:49.557746 13300 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:49.558068 13300 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:49.559233 6737 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2]). Expected version 1 got 0
I20240501 07:58:49.563633 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2] alter complete (version 1)
I20240501 07:58:49.597916 12825 raft_consensus.cc:482] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:49.598033 12825 raft_consensus.cc:504] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.598489 12825 leader_election.cc:288] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:49.598873 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba12580263f4bd883ad7e003ebd6c51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:49.599035 9666 raft_consensus.cc:2454] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:49.598984 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba12580263f4bd883ad7e003ebd6c51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:49.599154 6876 raft_consensus.cc:2454] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:49.599313 7207 leader_election.cc:302] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:49.599501 12825 raft_consensus.cc:2781] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:49.599587 12825 raft_consensus.cc:482] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:49.599637 12825 raft_consensus.cc:3037] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.600548 12825 raft_consensus.cc:504] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.600863 12825 leader_election.cc:288] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:49.601295 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba12580263f4bd883ad7e003ebd6c51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:49.601442 9666 raft_consensus.cc:3037] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.601466 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba12580263f4bd883ad7e003ebd6c51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:49.601576 6876 raft_consensus.cc:3037] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.602480 9666 raft_consensus.cc:2454] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:49.602659 6876 raft_consensus.cc:2454] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:49.602767 7207 leader_election.cc:302] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:49.602957 12825 raft_consensus.cc:2781] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:49.603169 12825 raft_consensus.cc:686] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:49.603368 12825 consensus_queue.cc:237] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:49.605326 6730 catalog_manager.cc:5484] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:49.637827 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.641707 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.643323 9646 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:49.647511 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:49.653901 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:49.654263 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:49.663759 9382 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:49.666462 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:49.678179 13299 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:49.678483 13299 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:49.678929 13298 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:49.679795 13298 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:49.680943 13300 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:49.681285 13300 tablet.cc:1620] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:49.685518 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.689953 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=27365dcedb074073ae837f62866a5ad2] alter complete (version 2)
I20240501 07:58:49.736934 7271 consensus_queue.cc:237] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:49.738270 9535 raft_consensus.cc:1261] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.738310 9271 raft_consensus.cc:1261] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.738679 13301 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.738796 12825 consensus_queue.cc:1035] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.740703 13333 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:49.741086 9271 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:49.742342 9535 raft_consensus.cc:2932] T 22cb0ec763ff4941acbfbb007995241d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:49.744087 6737 catalog_manager.cc:5484] T 22cb0ec763ff4941acbfbb007995241d P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:49.747041 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 22cb0ec763ff4941acbfbb007995241d with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:49.747473 13338 tablet_replica.cc:330] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:49.747620 13338 raft_consensus.cc:2229] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.747754 13338 raft_consensus.cc:2258] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.748379 13338 ts_tablet_manager.cc:1875] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:49.750170 13338 ts_tablet_manager.cc:1888] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:49.750238 13338 log.cc:1196] T 22cb0ec763ff4941acbfbb007995241d P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/22cb0ec763ff4941acbfbb007995241d
I20240501 07:58:49.750849 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 22cb0ec763ff4941acbfbb007995241d (table rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]) successfully deleted
I20240501 07:58:49.797251 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:49.804296 13298 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:49.808370 13297 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.808641 13300 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:49.810233 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=27365dcedb074073ae837f62866a5ad2] alter complete (version 3)
I20240501 07:58:49.811257 13299 tablet.cc:1722] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:49.816434 11060 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.855366 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:49.860468 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.863320 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:49.866364 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:49.869727 9646 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:49.870100 9382 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:49.871800 12648 consensus_queue.cc:1035] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.883868 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.885053 12648 consensus_queue.cc:1035] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.889647 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:49.915984 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:49.916175 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:49.918431 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 774f9074f2b24c78bed9a4d705d2532f
I20240501 07:58:49.919026 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 774f9074f2b24c78bed9a4d705d2532f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:49.919135 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet 774f9074f2b24c78bed9a4d705d2532f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:49.919234 13338 tablet_replica.cc:330] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:49.919360 13338 raft_consensus.cc:2229] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:49.919476 13408 tablet_replica.cc:330] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:49.919540 13338 raft_consensus.cc:2258] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.919757 13408 raft_consensus.cc:2229] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.919919 13408 raft_consensus.cc:2258] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.920120 13338 ts_tablet_manager.cc:1875] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.920305 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet 774f9074f2b24c78bed9a4d705d2532f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:49 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:49.920771 13408 ts_tablet_manager.cc:1875] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.922335 13338 ts_tablet_manager.cc:1888] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.922398 13338 log.cc:1196] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/774f9074f2b24c78bed9a4d705d2532f
I20240501 07:58:49.922673 13338 ts_tablet_manager.cc:1909] T 774f9074f2b24c78bed9a4d705d2532f P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:49.923195 13409 tablet_replica.cc:330] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:49.923399 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table_new_name [id=27365dcedb074073ae837f62866a5ad2]) successfully deleted
I20240501 07:58:49.923446 13409 raft_consensus.cc:2229] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.923920 13408 ts_tablet_manager.cc:1888] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.924044 13408 log.cc:1196] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/774f9074f2b24c78bed9a4d705d2532f
I20240501 07:58:49.924379 13408 ts_tablet_manager.cc:1909] T 774f9074f2b24c78bed9a4d705d2532f P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:49.923681 13409 raft_consensus.cc:2258] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.925071 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table_new_name [id=27365dcedb074073ae837f62866a5ad2]) successfully deleted
I20240501 07:58:49.925525 13409 ts_tablet_manager.cc:1875] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:49.927453 13409 ts_tablet_manager.cc:1888] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:49.927520 13409 log.cc:1196] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/774f9074f2b24c78bed9a4d705d2532f
I20240501 07:58:49.927775 13409 ts_tablet_manager.cc:1909] T 774f9074f2b24c78bed9a4d705d2532f P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:49.928364 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 774f9074f2b24c78bed9a4d705d2532f (table rebalancer_dynamic_table_new_name [id=27365dcedb074073ae837f62866a5ad2]) successfully deleted
I20240501 07:58:49.959422 9535 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:49.960623 7140 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.960645 9666 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:49.961024 11763 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.961179 11487 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:49.962989 13428 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:49.963342 9666 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:49.963366 7140 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:49.965935 6733 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:49.969110 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet f8921a8351e04daba09a8e7e4f683fd9 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:49.969369 13409 tablet_replica.cc:330] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:49.969534 13409 raft_consensus.cc:2229] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.969671 13409 raft_consensus.cc:2258] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.970078 13409 ts_tablet_manager.cc:1875] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:49.972033 13409 ts_tablet_manager.cc:1888] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:49.972095 13409 log.cc:1196] T f8921a8351e04daba09a8e7e4f683fd9 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f8921a8351e04daba09a8e7e4f683fd9
I20240501 07:58:49.972545 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet f8921a8351e04daba09a8e7e4f683fd9 (table rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]) successfully deleted
I20240501 07:58:49.989231 13301 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:49.993067 13301 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.023527 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:50.029433 7121 tablet_service.cc:1467] Processing CreateTablet for tablet fd13bd632a814bfc89421ddf02906b4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]), partition=
I20240501 07:58:50.029714 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd13bd632a814bfc89421ddf02906b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.029800 9514 tablet_service.cc:1467] Processing CreateTablet for tablet fd13bd632a814bfc89421ddf02906b4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]), partition=
I20240501 07:58:50.030072 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd13bd632a814bfc89421ddf02906b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.030090 9382 tablet_service.cc:1467] Processing CreateTablet for tablet fd13bd632a814bfc89421ddf02906b4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]), partition=
I20240501 07:58:50.030328 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd13bd632a814bfc89421ddf02906b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.031742 13474 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:50.032742 13474 tablet_bootstrap.cc:654] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.033854 13474 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:50.034022 13474 ts_tablet_manager.cc:1367] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:50.034546 13476 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.034550 13474 raft_consensus.cc:348] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.034677 13474 raft_consensus.cc:374] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.034729 13474 raft_consensus.cc:729] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.034911 13474 consensus_queue.cc:260] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.035295 13474 ts_tablet_manager.cc:1398] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.039803 13476 tablet_bootstrap.cc:654] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.040556 13475 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:50.042660 13475 tablet_bootstrap.cc:654] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.046119 13475 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:50.046279 13475 ts_tablet_manager.cc:1367] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:50.047288 13476 tablet_bootstrap.cc:492] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.047389 13476 ts_tablet_manager.cc:1367] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:50.047361 13475 raft_consensus.cc:348] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.047472 13475 raft_consensus.cc:374] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.047505 13475 raft_consensus.cc:729] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.047617 13475 consensus_queue.cc:260] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.047874 13476 raft_consensus.cc:348] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.047948 13475 ts_tablet_manager.cc:1398] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.047982 13476 raft_consensus.cc:374] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.048017 13476 raft_consensus.cc:729] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.048143 13476 consensus_queue.cc:260] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.048553 13476 ts_tablet_manager.cc:1398] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:50.087539 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.087544 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.088163 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.095675 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.096127 9514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:50.103447 9382 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.109174 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.116596 9646 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:50.120157 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:50.126695 7252 tablet_service.cc:1467] Processing CreateTablet for tablet a7b180ca614d4eaf812ae2cf7485b715 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.127089 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b180ca614d4eaf812ae2cf7485b715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.127566 9646 tablet_service.cc:1467] Processing CreateTablet for tablet c684b44a722143c4a3c44caad7cd84be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.127740 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 79ab7f97eb2e439f8c974636b5791824 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.127827 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684b44a722143c4a3c44caad7cd84be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.127774 9645 tablet_service.cc:1467] Processing CreateTablet for tablet a7b180ca614d4eaf812ae2cf7485b715 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.127954 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ab7f97eb2e439f8c974636b5791824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.128206 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b180ca614d4eaf812ae2cf7485b715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.129686 6989 tablet_service.cc:1467] Processing CreateTablet for tablet c684b44a722143c4a3c44caad7cd84be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.129961 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684b44a722143c4a3c44caad7cd84be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.129905 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 79ab7f97eb2e439f8c974636b5791824 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.130156 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ab7f97eb2e439f8c974636b5791824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.130955 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 79ab7f97eb2e439f8c974636b5791824 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.131264 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ab7f97eb2e439f8c974636b5791824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.132293 9381 tablet_service.cc:1467] Processing CreateTablet for tablet a7b180ca614d4eaf812ae2cf7485b715 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.132498 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b180ca614d4eaf812ae2cf7485b715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.133136 13475 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:50.133270 13476 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.134232 13476 tablet_bootstrap.cc:654] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.134232 13475 tablet_bootstrap.cc:654] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.135756 9382 tablet_service.cc:1467] Processing CreateTablet for tablet c684b44a722143c4a3c44caad7cd84be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.135955 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684b44a722143c4a3c44caad7cd84be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.138911 13475 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:50.139035 13475 ts_tablet_manager.cc:1367] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 07:58:50.139472 13475 raft_consensus.cc:348] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.139575 13475 raft_consensus.cc:374] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.139611 13475 raft_consensus.cc:729] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.139636 13505 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:50.139719 13475 consensus_queue.cc:260] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.142293 13475 ts_tablet_manager.cc:1398] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:50.142871 13505 tablet_bootstrap.cc:654] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.143960 13476 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.144052 13476 ts_tablet_manager.cc:1367] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 07:58:50.144501 13476 raft_consensus.cc:348] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.144598 13476 raft_consensus.cc:374] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.144629 13476 raft_consensus.cc:729] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.144732 13476 consensus_queue.cc:260] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.145025 13476 ts_tablet_manager.cc:1398] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.145516 13476 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.146409 13476 tablet_bootstrap.cc:654] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.151197 13505 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:50.151283 13476 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.151326 13505 ts_tablet_manager.cc:1367] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:50.151364 13476 ts_tablet_manager.cc:1367] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:50.151775 13505 raft_consensus.cc:348] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.151784 13476 raft_consensus.cc:348] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.151880 13476 raft_consensus.cc:374] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.151883 13505 raft_consensus.cc:374] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.151906 13476 raft_consensus.cc:729] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.151949 13505 raft_consensus.cc:729] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.152024 13476 consensus_queue.cc:260] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.152089 13505 consensus_queue.cc:260] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.152331 13476 ts_tablet_manager.cc:1398] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.152577 13476 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.153221 13505 ts_tablet_manager.cc:1398] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.153434 13505 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:50.154423 13476 tablet_bootstrap.cc:654] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.155382 13505 tablet_bootstrap.cc:654] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.156634 13505 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:50.156764 13505 ts_tablet_manager.cc:1367] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:50.157292 13505 raft_consensus.cc:348] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.157450 13505 raft_consensus.cc:374] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.157502 13505 raft_consensus.cc:729] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.157646 13505 consensus_queue.cc:260] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.158499 13505 ts_tablet_manager.cc:1398] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.158669 13505 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:50.159448 13505 tablet_bootstrap.cc:654] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.161343 13505 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:50.161473 13505 ts_tablet_manager.cc:1367] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:50.162206 13505 raft_consensus.cc:348] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.162328 13505 raft_consensus.cc:374] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.162370 13505 raft_consensus.cc:729] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.162494 13505 consensus_queue.cc:260] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.162778 13505 ts_tablet_manager.cc:1398] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.166335 13506 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:50.167273 13504 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:50.167928 13506 tablet_bootstrap.cc:654] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.168745 13504 tablet_bootstrap.cc:654] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.169301 13506 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:50.169384 13506 ts_tablet_manager.cc:1367] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240501 07:58:50.169797 13506 raft_consensus.cc:348] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.169916 13506 raft_consensus.cc:374] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.169945 13506 raft_consensus.cc:729] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.170061 13506 consensus_queue.cc:260] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.170312 13506 ts_tablet_manager.cc:1398] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:50.180156 13476 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.180259 13476 ts_tablet_manager.cc:1367] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 07:58:50.180739 13504 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:50.180697 13476 raft_consensus.cc:348] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.180799 13476 raft_consensus.cc:374] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.180828 13476 raft_consensus.cc:729] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.180833 13504 ts_tablet_manager.cc:1367] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240501 07:58:50.180939 13476 consensus_queue.cc:260] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.181190 13476 ts_tablet_manager.cc:1398] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:50.181324 13504 raft_consensus.cc:348] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.181430 13504 raft_consensus.cc:374] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.181461 13504 raft_consensus.cc:729] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.181569 13504 consensus_queue.cc:260] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.183209 13504 ts_tablet_manager.cc:1398] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:50.210702 7140 consensus_queue.cc:237] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.211972 9535 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:50.211975 6877 raft_consensus.cc:1261] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:50.212386 13429 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:50.212561 12756 consensus_queue.cc:1035] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:50.214413 13429 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.214772 12912 raft_consensus.cc:482] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:50.214869 12912 raft_consensus.cc:504] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.215667 12912 leader_election.cc:288] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:50.215984 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ab7f97eb2e439f8c974636b5791824" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:50.216094 9534 raft_consensus.cc:2454] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:50.216383 9339 leader_election.cc:302] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.216405 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ab7f97eb2e439f8c974636b5791824" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:50.216518 9666 raft_consensus.cc:2454] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:50.216552 12912 raft_consensus.cc:2781] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:50.216604 12912 raft_consensus.cc:482] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:50.216639 12912 raft_consensus.cc:3037] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.217521 12912 raft_consensus.cc:504] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.217878 12912 leader_election.cc:288] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:50.218683 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ab7f97eb2e439f8c974636b5791824" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:50.218783 9666 raft_consensus.cc:3037] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.219484 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ab7f97eb2e439f8c974636b5791824" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:50.219590 9534 raft_consensus.cc:3037] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.219959 9666 raft_consensus.cc:2454] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:50.220468 6731 catalog_manager.cc:5484] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.220608 9534 raft_consensus.cc:2454] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:50.220855 9339 leader_election.cc:302] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.221208 13297 raft_consensus.cc:2781] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.221300 13297 raft_consensus.cc:686] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:50.221433 13297 consensus_queue.cc:237] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:50.221518 6877 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.221747 9534 raft_consensus.cc:2932] T f723d182d04d439da6ab5b811448cf2d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.223608 6734 catalog_manager.cc:5484] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:50.225982 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet f723d182d04d439da6ab5b811448cf2d with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:50.226415 13540 tablet_replica.cc:330] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:50.226559 13540 raft_consensus.cc:2229] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.226732 13540 raft_consensus.cc:2258] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.227389 13540 ts_tablet_manager.cc:1875] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:50.229377 13540 ts_tablet_manager.cc:1888] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:50.229455 13540 log.cc:1196] T f723d182d04d439da6ab5b811448cf2d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f723d182d04d439da6ab5b811448cf2d
I20240501 07:58:50.229979 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet f723d182d04d439da6ab5b811448cf2d (table rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]) successfully deleted
I20240501 07:58:50.278617 12574 raft_consensus.cc:482] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:50.278734 12574 raft_consensus.cc:504] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.279172 12574 leader_election.cc:288] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:50.279466 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b180ca614d4eaf812ae2cf7485b715" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:50.279620 9402 raft_consensus.cc:2454] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:50.279683 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b180ca614d4eaf812ae2cf7485b715" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:50.279868 7271 raft_consensus.cc:2454] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:50.280050 9601 leader_election.cc:302] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.280206 12574 raft_consensus.cc:2781] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:50.280256 12574 raft_consensus.cc:482] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:50.280297 12574 raft_consensus.cc:3037] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.281340 12574 raft_consensus.cc:504] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.281651 12574 leader_election.cc:288] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:50.281970 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b180ca614d4eaf812ae2cf7485b715" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:50.281968 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b180ca614d4eaf812ae2cf7485b715" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:50.282177 9402 raft_consensus.cc:3037] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.282177 7271 raft_consensus.cc:3037] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.283301 9402 raft_consensus.cc:2454] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:50.283372 7271 raft_consensus.cc:2454] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:50.283677 9601 leader_election.cc:302] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.283869 12574 raft_consensus.cc:2781] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.283949 12574 raft_consensus.cc:686] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:50.284094 12574 consensus_queue.cc:237] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.285785 13428 raft_consensus.cc:482] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:50.285877 13428 raft_consensus.cc:504] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.286257 13428 leader_election.cc:288] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:50.286540 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd13bd632a814bfc89421ddf02906b4b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:50.286646 9402 raft_consensus.cc:2454] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:50.286742 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd13bd632a814bfc89421ddf02906b4b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:50.286895 7140 raft_consensus.cc:2454] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:50.287144 9470 leader_election.cc:302] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.287600 13428 raft_consensus.cc:2781] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:50.287658 13428 raft_consensus.cc:482] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:50.287704 13428 raft_consensus.cc:3037] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.288702 13428 raft_consensus.cc:504] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.289067 13428 leader_election.cc:288] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:50.289316 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd13bd632a814bfc89421ddf02906b4b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:50.289366 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd13bd632a814bfc89421ddf02906b4b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:50.289433 7140 raft_consensus.cc:3037] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.289463 9402 raft_consensus.cc:3037] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.290388 9402 raft_consensus.cc:2454] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:50.290462 7140 raft_consensus.cc:2454] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:50.290764 9469 leader_election.cc:302] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:50.291110 13428 raft_consensus.cc:2781] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.291206 13428 raft_consensus.cc:686] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:50.291332 13428 consensus_queue.cc:237] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.292610 6733 catalog_manager.cc:5484] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:50.293452 6732 catalog_manager.cc:5484] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.314940 12574 raft_consensus.cc:482] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:50.315091 12574 raft_consensus.cc:504] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.315577 12574 leader_election.cc:288] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:50.315872 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684b44a722143c4a3c44caad7cd84be" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:50.316022 9402 raft_consensus.cc:2454] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:50.316023 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684b44a722143c4a3c44caad7cd84be" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:50.316156 7008 raft_consensus.cc:2454] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:50.316326 9601 leader_election.cc:302] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.316479 13577 raft_consensus.cc:2781] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:50.316540 13577 raft_consensus.cc:482] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:50.316581 13577 raft_consensus.cc:3037] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.317548 13577 raft_consensus.cc:504] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.317875 13577 leader_election.cc:288] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:50.318171 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684b44a722143c4a3c44caad7cd84be" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:50.318291 9402 raft_consensus.cc:3037] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.318327 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684b44a722143c4a3c44caad7cd84be" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:50.318455 7008 raft_consensus.cc:3037] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.319406 9402 raft_consensus.cc:2454] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:50.319636 7008 raft_consensus.cc:2454] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:50.319743 9601 leader_election.cc:302] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.319916 13577 raft_consensus.cc:2781] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.319981 13577 raft_consensus.cc:686] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:50.320124 13577 consensus_queue.cc:237] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:50.322139 6734 catalog_manager.cc:5484] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:50.341495 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.345338 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.351769 6859 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.352979 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.364811 9382 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.365484 9514 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:50.366143 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.402065 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.437328 7008 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd"
dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52662
I20240501 07:58:50.437475 7008 raft_consensus.cc:595] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:50.445294 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:50.451174 9402 raft_consensus.cc:1261] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:50.451215 7140 raft_consensus.cc:1261] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:50.451666 11487 consensus_queue.cc:1035] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.451987 11487 consensus_queue.cc:1035] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.455360 6734 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]). Expected version 1 got 0
I20240501 07:58:50.456131 6730 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]). Expected version 1 got 0
I20240501 07:58:50.456184 13609 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:50.456661 13609 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:50.458070 12648 raft_consensus.cc:979] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: : Instructing follower 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:58:50.458174 12648 raft_consensus.cc:1067] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Signalling peer 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:58:50.458181 13610 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:50.458536 13610 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:50.458689 6877 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd"
dest_uuid: "1a9029bee6a144938f68b363e4cca162"
from {username='slave'} at 127.0.99.130:56913
I20240501 07:58:50.458855 6877 raft_consensus.cc:482] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:50.458921 6877 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:50.459174 6730 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]). Expected version 1 got 0
I20240501 07:58:50.459995 6877 raft_consensus.cc:504] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.460498 6877 leader_election.cc:288] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:50.460736 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:50.460860 7008 raft_consensus.cc:3032] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:50.460901 7008 raft_consensus.cc:729] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:50.461046 7008 consensus_queue.cc:260] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.461217 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:50.461266 7008 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:50.461330 9534 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:50.461356 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69ff85c89fa443539b28c0d0cc299cfd" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:50.461452 7140 raft_consensus.cc:3037] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:50.462262 9534 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:58:50.462317 7140 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:58:50.462342 7008 raft_consensus.cc:2454] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:58:50.462658 6815 leader_election.cc:302] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:50.462839 13314 raft_consensus.cc:2781] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:50.462918 13314 raft_consensus.cc:686] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:50.463140 13314 consensus_queue.cc:237] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.465644 6733 catalog_manager.cc:5484] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:50.466599 13612 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:50.466975 13612 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:50.467912 6729 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773]). Expected version 1 got 0
I20240501 07:58:50.468394 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773] alter complete (version 1)
I20240501 07:58:50.543527 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.553295 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.556383 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.571655 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.572289 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:50.573557 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.579625 13612 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:50.579946 13612 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:50.580339 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.581337 13609 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:50.581645 13609 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:50.585492 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=6679f9c825d04cd889b0c0765d51a773] alter complete (version 2)
I20240501 07:58:50.593287 13610 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:50.593619 13610 tablet.cc:1620] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:50.593992 9514 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:50.594796 9382 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.616192 11060 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.619916 13297 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.691179 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:50.698895 13609 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:50.699952 13612 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:50.702363 13610 tablet.cc:1722] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:50.707850 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=6679f9c825d04cd889b0c0765d51a773] alter complete (version 3)
I20240501 07:58:50.727494 13577 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.733508 12574 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.745497 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.745575 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.753849 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.754467 9251 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.755632 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:50.771929 9514 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:50.772272 9382 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:50.803824 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:50.811321 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:50.811489 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:50.813650 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet fd13bd632a814bfc89421ddf02906b4b
I20240501 07:58:50.814127 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet fd13bd632a814bfc89421ddf02906b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:50 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:50.814662 9514 tablet_service.cc:1514] Processing DeleteTablet for tablet fd13bd632a814bfc89421ddf02906b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:50 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:50.815119 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet fd13bd632a814bfc89421ddf02906b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:50 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:50.815403 13715 tablet_replica.cc:330] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:50.815605 13715 raft_consensus.cc:2229] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.815728 13714 tablet_replica.cc:330] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:50.815796 13715 raft_consensus.cc:2258] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.815919 13714 raft_consensus.cc:2229] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:50.816145 13714 raft_consensus.cc:2258] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.816499 13715 ts_tablet_manager.cc:1875] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:50.816669 13714 ts_tablet_manager.cc:1875] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:50.818367 13713 tablet_replica.cc:330] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:50.818563 13713 raft_consensus.cc:2229] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.819090 13714 ts_tablet_manager.cc:1888] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:50.819240 13714 log.cc:1196] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/fd13bd632a814bfc89421ddf02906b4b
I20240501 07:58:50.819264 13715 ts_tablet_manager.cc:1888] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:50.819455 13715 log.cc:1196] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/fd13bd632a814bfc89421ddf02906b4b
I20240501 07:58:50.819640 13714 ts_tablet_manager.cc:1909] T fd13bd632a814bfc89421ddf02906b4b P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:50.819790 13715 ts_tablet_manager.cc:1909] T fd13bd632a814bfc89421ddf02906b4b P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:50.820492 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table_new_name [id=6679f9c825d04cd889b0c0765d51a773]) successfully deleted
I20240501 07:58:50.820612 13713 raft_consensus.cc:2258] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.821069 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table_new_name [id=6679f9c825d04cd889b0c0765d51a773]) successfully deleted
I20240501 07:58:50.821733 13713 ts_tablet_manager.cc:1875] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:50.823637 13713 ts_tablet_manager.cc:1888] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:50.823704 13713 log.cc:1196] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd13bd632a814bfc89421ddf02906b4b
I20240501 07:58:50.824041 13713 ts_tablet_manager.cc:1909] T fd13bd632a814bfc89421ddf02906b4b P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:50.824690 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet fd13bd632a814bfc89421ddf02906b4b (table rebalancer_dynamic_table_new_name [id=6679f9c825d04cd889b0c0765d51a773]) successfully deleted
I20240501 07:58:50.851178 12574 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.852661 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:50.856873 12574 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:50.857591 7271 consensus_queue.cc:237] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:50.859375 9402 raft_consensus.cc:1261] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:50.859724 6877 raft_consensus.cc:1261] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:50.859722 13301 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:50.860419 13301 consensus_queue.cc:1035] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:50.863036 13301 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:50.863962 9514 tablet_service.cc:1467] Processing CreateTablet for tablet 4aea3c6384b74a32a7605f9777534618 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.864092 7252 tablet_service.cc:1467] Processing CreateTablet for tablet d3333244e06a4ad8bd3ecc73cb356c90 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.864199 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aea3c6384b74a32a7605f9777534618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.864370 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3333244e06a4ad8bd3ecc73cb356c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.866500 9507 tablet_service.cc:1467] Processing CreateTablet for tablet d3333244e06a4ad8bd3ecc73cb356c90 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.866747 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3333244e06a4ad8bd3ecc73cb356c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.866772 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4af3cf8f044e2be0f8e3195f8c084 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.866878 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4af3cf8f044e2be0f8e3195f8c084 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.866956 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4af3cf8f044e2be0f8e3195f8c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.867197 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4af3cf8f044e2be0f8e3195f8c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.867368 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 4aea3c6384b74a32a7605f9777534618 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.867574 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aea3c6384b74a32a7605f9777534618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.867862 6735 catalog_manager.cc:5484] T 8d4ea1beadd542a894f002ae78d3b37f P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:50.869555 9381 tablet_service.cc:1467] Processing CreateTablet for tablet d3333244e06a4ad8bd3ecc73cb356c90 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.869731 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4af3cf8f044e2be0f8e3195f8c084 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.870174 13738 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:50.871481 9401 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:50.871846 13738 tablet_bootstrap.cc:654] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.869555 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 4aea3c6384b74a32a7605f9777534618 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:50.873080 13738 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:50.873168 13738 ts_tablet_manager.cc:1367] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:50.873657 13738 raft_consensus.cc:348] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.873761 13738 raft_consensus.cc:374] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.873795 13738 raft_consensus.cc:729] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.873886 13738 consensus_queue.cc:260] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.874178 13738 ts_tablet_manager.cc:1398] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.874347 13738 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:50.874547 7138 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:50.869798 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3333244e06a4ad8bd3ecc73cb356c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.875299 13738 tablet_bootstrap.cc:654] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.875494 12885 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:50.877048 13743 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.877080 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4af3cf8f044e2be0f8e3195f8c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.877245 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aea3c6384b74a32a7605f9777534618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.878964 6877 raft_consensus.cc:2932] T 8d4ea1beadd542a894f002ae78d3b37f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:50.879765 13738 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:50.879845 13738 ts_tablet_manager.cc:1367] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:50.880435 13738 raft_consensus.cc:348] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.880532 13738 raft_consensus.cc:374] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.880564 13738 raft_consensus.cc:729] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.880692 13738 consensus_queue.cc:260] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.881016 13738 ts_tablet_manager.cc:1398] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.881193 13738 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:50.882036 13740 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:50.882637 13738 tablet_bootstrap.cc:654] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.882656 13739 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:50.882946 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4ea1beadd542a894f002ae78d3b37f with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:50.883374 13749 tablet_replica.cc:330] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:50.883522 13749 raft_consensus.cc:2229] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.883647 13739 tablet_bootstrap.cc:654] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.883702 13749 raft_consensus.cc:2258] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.884142 13749 ts_tablet_manager.cc:1875] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:50.886479 13743 tablet_bootstrap.cc:654] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.886512 13749 ts_tablet_manager.cc:1888] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:50.886644 13740 tablet_bootstrap.cc:654] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.886711 13749 log.cc:1196] T 8d4ea1beadd542a894f002ae78d3b37f P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8d4ea1beadd542a894f002ae78d3b37f
I20240501 07:58:50.887692 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 8d4ea1beadd542a894f002ae78d3b37f (table rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]) successfully deleted
I20240501 07:58:50.888203 13738 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:50.888294 13738 ts_tablet_manager.cc:1367] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:50.888767 13738 raft_consensus.cc:348] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.888880 13738 raft_consensus.cc:374] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.888911 13738 raft_consensus.cc:729] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.889043 13740 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:50.889042 13738 consensus_queue.cc:260] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.889150 13740 ts_tablet_manager.cc:1367] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:50.889222 13739 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:50.889300 13739 ts_tablet_manager.cc:1367] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:50.889323 13738 ts_tablet_manager.cc:1398] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.889706 13740 raft_consensus.cc:348] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.889837 13740 raft_consensus.cc:374] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.889876 13740 raft_consensus.cc:729] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.889768 13739 raft_consensus.cc:348] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.889956 13739 raft_consensus.cc:374] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.889995 13739 raft_consensus.cc:729] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.890014 13740 consensus_queue.cc:260] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.890107 13739 consensus_queue.cc:260] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.890336 13740 ts_tablet_manager.cc:1398] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:50.890355 13739 ts_tablet_manager.cc:1398] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.890532 13740 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:50.891741 13740 tablet_bootstrap.cc:654] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.892270 13743 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.892390 13743 ts_tablet_manager.cc:1367] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240501 07:58:50.892949 13743 raft_consensus.cc:348] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.893018 13740 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:50.893113 13740 ts_tablet_manager.cc:1367] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:50.893136 13743 raft_consensus.cc:374] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.893204 13743 raft_consensus.cc:729] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.893388 13743 consensus_queue.cc:260] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.893740 13740 raft_consensus.cc:348] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.893909 13740 raft_consensus.cc:374] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.894003 13740 raft_consensus.cc:729] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.894181 13740 consensus_queue.cc:260] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.894486 13743 ts_tablet_manager.cc:1398] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:50.894634 13743 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.894840 13740 ts_tablet_manager.cc:1398] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.896435 13743 tablet_bootstrap.cc:654] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.898176 13743 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.898315 13743 ts_tablet_manager.cc:1367] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:50.898957 13743 raft_consensus.cc:348] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.899199 13743 raft_consensus.cc:374] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.899286 13743 raft_consensus.cc:729] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.899464 13743 consensus_queue.cc:260] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:50.900173 13743 ts_tablet_manager.cc:1398] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.900431 13743 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:50.901548 13743 tablet_bootstrap.cc:654] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.902916 13743 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:50.903028 13743 ts_tablet_manager.cc:1367] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:50.903478 13743 raft_consensus.cc:348] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.903597 13743 raft_consensus.cc:374] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.903635 13743 raft_consensus.cc:729] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.903771 13743 consensus_queue.cc:260] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.904111 13743 ts_tablet_manager.cc:1398] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:50.915130 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:50.920512 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]), partition=
I20240501 07:58:50.920809 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a02367c4c7a4e229af38dc27571fec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.920948 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]), partition=
I20240501 07:58:50.921051 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]), partition=
I20240501 07:58:50.921157 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a02367c4c7a4e229af38dc27571fec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.921276 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a02367c4c7a4e229af38dc27571fec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.923243 13740 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:50.924114 13765 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:50.924729 13740 tablet_bootstrap.cc:654] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.925387 13764 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:50.925441 13765 tablet_bootstrap.cc:654] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.926281 13740 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:50.926373 13740 ts_tablet_manager.cc:1367] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:50.926896 13740 raft_consensus.cc:348] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.927042 13740 raft_consensus.cc:374] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.927079 13740 raft_consensus.cc:729] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.927182 13765 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:50.927202 13740 consensus_queue.cc:260] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.927259 13765 ts_tablet_manager.cc:1367] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:50.927443 13764 tablet_bootstrap.cc:654] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:50.927510 13740 ts_tablet_manager.cc:1398] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.927695 13765 raft_consensus.cc:348] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.927806 13765 raft_consensus.cc:374] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.927836 13765 raft_consensus.cc:729] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.927946 13765 consensus_queue.cc:260] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.928189 13765 ts_tablet_manager.cc:1398] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.928824 13764 tablet_bootstrap.cc:492] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:50.930004 13764 ts_tablet_manager.cc:1367] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:50.930553 13764 raft_consensus.cc:348] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.930752 13764 raft_consensus.cc:374] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:50.930842 13764 raft_consensus.cc:729] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:50.931089 13764 consensus_queue.cc:260] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:50.931622 13764 ts_tablet_manager.cc:1398] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:50.934937 9534 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:50.935382 12885 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:50.938314 7008 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:50.938333 13297 raft_consensus.cc:482] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:50.938434 13297 raft_consensus.cc:504] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.938946 13297 leader_election.cc:288] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:50.940480 12885 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:50.940894 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3333244e06a4ad8bd3ecc73cb356c90" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:50.941025 9534 raft_consensus.cc:2454] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:50.941368 9339 leader_election.cc:302] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.941547 13297 raft_consensus.cc:2781] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:50.941596 13297 raft_consensus.cc:482] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:50.941633 13297 raft_consensus.cc:3037] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.942719 13297 raft_consensus.cc:504] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.943101 13297 leader_election.cc:288] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:50.944459 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3333244e06a4ad8bd3ecc73cb356c90" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:50.944572 9534 raft_consensus.cc:3037] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:50.945605 9534 raft_consensus.cc:2454] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:50.945956 9339 leader_election.cc:302] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:50.946465 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3333244e06a4ad8bd3ecc73cb356c90" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:50.946503 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3333244e06a4ad8bd3ecc73cb356c90" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:50.946611 7271 raft_consensus.cc:2454] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:50.946799 13297 raft_consensus.cc:2781] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:50.946884 13297 raft_consensus.cc:686] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:50.947044 13297 consensus_queue.cc:237] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:50.948846 6731 catalog_manager.cc:5484] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:51.034842 11487 raft_consensus.cc:482] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.034984 11487 raft_consensus.cc:504] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.035408 11487 leader_election.cc:288] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:51.035867 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:51.036002 9271 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:51.036327 9470 leader_election.cc:302] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.036746 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:51.036883 9401 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:51.037446 11487 raft_consensus.cc:2781] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.037506 11487 raft_consensus.cc:482] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.037551 11487 raft_consensus.cc:3037] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.038522 11487 raft_consensus.cc:504] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.038784 11487 leader_election.cc:288] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:51.039151 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:51.039187 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:51.039260 9401 raft_consensus.cc:3037] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.039315 9271 raft_consensus.cc:3037] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.040177 9401 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:51.040262 9271 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:51.040517 9470 leader_election.cc:302] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.040673 11487 raft_consensus.cc:2781] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.040760 11487 raft_consensus.cc:686] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:51.040904 11487 consensus_queue.cc:237] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.043085 6734 catalog_manager.cc:5484] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:51.057056 11487 raft_consensus.cc:482] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.057176 11487 raft_consensus.cc:504] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:51.057552 11487 leader_election.cc:288] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:51.057945 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:51.058090 9271 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:51.058131 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:51.058291 9401 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:51.058534 9470 leader_election.cc:302] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.058706 11487 raft_consensus.cc:2781] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.058754 11487 raft_consensus.cc:482] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.058794 11487 raft_consensus.cc:3037] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.059935 11487 raft_consensus.cc:504] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:51.060230 11487 leader_election.cc:288] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:51.060559 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:51.060570 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:51.060662 9271 raft_consensus.cc:3037] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.060688 9401 raft_consensus.cc:3037] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.061633 9271 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:51.061702 9401 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:51.061954 9470 leader_election.cc:302] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.062103 11763 raft_consensus.cc:2781] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.062214 11763 raft_consensus.cc:686] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:51.062353 11763 consensus_queue.cc:237] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:51.064359 6732 catalog_manager.cc:5484] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:51.078141 12608 raft_consensus.cc:482] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.078265 12608 raft_consensus.cc:504] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.078770 12608 leader_election.cc:288] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:51.079155 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a02367c4c7a4e229af38dc27571fec1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:51.079165 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a02367c4c7a4e229af38dc27571fec1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:51.079293 6877 raft_consensus.cc:2454] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:51.079308 7138 raft_consensus.cc:2454] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:51.079597 9205 leader_election.cc:302] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:51.079813 12608 raft_consensus.cc:2781] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.079861 12608 raft_consensus.cc:482] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.079905 12608 raft_consensus.cc:3037] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.080773 12608 raft_consensus.cc:504] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.081123 12608 leader_election.cc:288] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:51.081470 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a02367c4c7a4e229af38dc27571fec1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:51.081619 6877 raft_consensus.cc:3037] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.081625 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a02367c4c7a4e229af38dc27571fec1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:51.081737 7138 raft_consensus.cc:3037] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.082655 7138 raft_consensus.cc:2454] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:51.082655 6877 raft_consensus.cc:2454] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:51.082922 9206 leader_election.cc:302] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:51.083164 12608 raft_consensus.cc:2781] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.083245 12608 raft_consensus.cc:686] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:51.083361 12608 consensus_queue.cc:237] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:51.085481 6734 catalog_manager.cc:5484] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:51.111331 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.113973 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.117681 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:51.118245 9251 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:51.118402 9507 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:51.125990 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.127173 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.151887 9383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.204169 7138 consensus_queue.cc:237] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:51.205592 9665 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:51.205912 7008 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:51.206312 13836 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:51.206544 12756 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:51.208443 13836 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:51.210475 7008 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:51.211126 9665 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:51.212829 6734 catalog_manager.cc:5484] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:51.215919 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:51.217069 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c30f6292754d97a839700be93f8c07 with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:51.217839 13845 tablet_replica.cc:330] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:51.219264 13845 raft_consensus.cc:2229] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:51.219506 13845 raft_consensus.cc:2258] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.220165 7138 raft_consensus.cc:1261] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:51.219878 6877 raft_consensus.cc:1261] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:51.220742 13808 consensus_queue.cc:1035] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.220885 13806 consensus_queue.cc:1035] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.221637 13845 ts_tablet_manager.cc:1875] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:51.223851 6733 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]). Expected version 1 got 0
I20240501 07:58:51.224442 6732 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]). Expected version 1 got 0
I20240501 07:58:51.225577 13845 ts_tablet_manager.cc:1888] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:51.225700 13845 log.cc:1196] T 82c30f6292754d97a839700be93f8c07 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/82c30f6292754d97a839700be93f8c07
I20240501 07:58:51.226502 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 82c30f6292754d97a839700be93f8c07 (table rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]) successfully deleted
I20240501 07:58:51.228332 6732 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]). Expected version 1 got 0
I20240501 07:58:51.229182 13850 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:51.229997 13850 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:51.231953 6732 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c]). Expected version 1 got 0
I20240501 07:58:51.232805 13849 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:51.233131 13849 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:51.237272 13857 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:51.237640 13857 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:51.238574 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c] alter complete (version 1)
I20240501 07:58:51.323627 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:51.325836 9251 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:51.327438 9383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.328075 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.331192 9507 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:51.331459 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.333703 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.344367 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:51.350497 13849 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:51.350808 13849 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:51.351083 13857 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:51.351333 13857 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:51.354863 13850 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:51.355155 13850 tablet.cc:1620] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:51.356715 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=22ac1fc0008d462e9520946b78c8462c] alter complete (version 2)
I20240501 07:58:51.376543 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.426728 9271 consensus_queue.cc:237] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:51.428031 9401 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:51.428035 9534 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:51.428419 13808 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:51.428640 13806 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:51.430598 13808 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:51.431097 9401 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:51.431198 9534 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:51.432304 11763 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.434077 6730 catalog_manager.cc:5484] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:51.438014 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5e1de6fc394fcb98c64165d74ca317 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:51.439450 13923 tablet_replica.cc:330] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:51.439937 13923 raft_consensus.cc:2229] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:51.440132 13923 raft_consensus.cc:2258] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.440641 13923 ts_tablet_manager.cc:1875] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:51.440680 11487 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.444147 13923 ts_tablet_manager.cc:1888] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:51.444252 13923 log.cc:1196] T 3c5e1de6fc394fcb98c64165d74ca317 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/3c5e1de6fc394fcb98c64165d74ca317
I20240501 07:58:51.444937 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 3c5e1de6fc394fcb98c64165d74ca317 (table rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]) successfully deleted
I20240501 07:58:51.460800 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:51.466382 13849 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:51.467391 13857 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:51.467754 13850 tablet.cc:1722] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:51.472271 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=22ac1fc0008d462e9520946b78c8462c] alter complete (version 3)
I20240501 07:58:51.481943 7270 raft_consensus.cc:3037] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.483223 12912 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.485956 13297 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.503684 11763 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.507046 11487 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.571547 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.572575 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:51.574249 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:51.574409 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:51.574559 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:51.583057 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 4a02367c4c7a4e229af38dc27571fec1
I20240501 07:58:51.583683 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:51 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:51.583945 13845 tablet_replica.cc:330] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:51.584100 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:51 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:51.584112 13845 raft_consensus.cc:2229] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:51.584255 13845 raft_consensus.cc:2258] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.586158 13995 tablet_replica.cc:330] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:51.586851 13995 raft_consensus.cc:2229] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:51.588030 13995 raft_consensus.cc:2258] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.589243 13995 ts_tablet_manager.cc:1875] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:51.591368 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.593227 9251 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a02367c4c7a4e229af38dc27571fec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:51 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:51.593583 7252 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.595415 13845 ts_tablet_manager.cc:1875] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:51.595463 13995 ts_tablet_manager.cc:1888] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:51.595518 13995 log.cc:1196] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a02367c4c7a4e229af38dc27571fec1
I20240501 07:58:51.595844 13995 ts_tablet_manager.cc:1909] T 4a02367c4c7a4e229af38dc27571fec1 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:51.597177 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table_new_name [id=22ac1fc0008d462e9520946b78c8462c]) successfully deleted
I20240501 07:58:51.597544 13845 ts_tablet_manager.cc:1888] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:51.597636 13845 log.cc:1196] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a02367c4c7a4e229af38dc27571fec1
I20240501 07:58:51.597987 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 4b683d8146a34fc6b74315f7db5d6890 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.598131 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 4b683d8146a34fc6b74315f7db5d6890 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.598261 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b683d8146a34fc6b74315f7db5d6890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.598290 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 3a54e29c093e4be485bf1af1e9ea7a72 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.599079 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 4b683d8146a34fc6b74315f7db5d6890 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.599330 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b683d8146a34fc6b74315f7db5d6890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.598367 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b683d8146a34fc6b74315f7db5d6890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.599917 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a54e29c093e4be485bf1af1e9ea7a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.600821 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 3a54e29c093e4be485bf1af1e9ea7a72 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.601109 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a54e29c093e4be485bf1af1e9ea7a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.601980 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 6a550051551640dcab9733948bb0966f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.598297 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 6a550051551640dcab9733948bb0966f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.602559 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a550051551640dcab9733948bb0966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.603500 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 3a54e29c093e4be485bf1af1e9ea7a72 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.599334 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 6a550051551640dcab9733948bb0966f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:51.603790 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a550051551640dcab9733948bb0966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.603837 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a54e29c093e4be485bf1af1e9ea7a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.605057 13845 ts_tablet_manager.cc:1909] T 4a02367c4c7a4e229af38dc27571fec1 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:51.605919 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a550051551640dcab9733948bb0966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.608011 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table_new_name [id=22ac1fc0008d462e9520946b78c8462c]) successfully deleted
I20240501 07:58:51.612068 9250 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:51.612562 14000 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:51.613600 14000 tablet_bootstrap.cc:654] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.616084 9383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.621619 6989 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:51.624919 9515 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:51.647214 13999 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:51.647502 13997 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:51.648572 13997 tablet_bootstrap.cc:654] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.649507 13999 tablet_bootstrap.cc:654] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.649749 14001 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:51.651166 13996 tablet_replica.cc:330] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:51.651521 13996 raft_consensus.cc:2229] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:51.652320 13996 raft_consensus.cc:2258] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.651391 13998 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:51.653438 13998 tablet_bootstrap.cc:654] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.659191 13996 ts_tablet_manager.cc:1875] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:51.659991 13998 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:51.660130 13998 ts_tablet_manager.cc:1367] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:51.660794 13998 raft_consensus.cc:348] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.660933 13998 raft_consensus.cc:374] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.660979 13998 raft_consensus.cc:729] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.661152 13998 consensus_queue.cc:260] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.661257 13996 ts_tablet_manager.cc:1888] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:51.661387 13996 log.cc:1196] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4a02367c4c7a4e229af38dc27571fec1
I20240501 07:58:51.661567 13998 ts_tablet_manager.cc:1398] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.661666 13996 ts_tablet_manager.cc:1909] T 4a02367c4c7a4e229af38dc27571fec1 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:51.662642 13998 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:51.664027 13998 tablet_bootstrap.cc:654] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.667593 13998 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:51.667673 13998 ts_tablet_manager.cc:1367] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:51.668085 13998 raft_consensus.cc:348] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.668191 13998 raft_consensus.cc:374] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.668222 13998 raft_consensus.cc:729] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.668352 13998 consensus_queue.cc:260] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.668635 13998 ts_tablet_manager.cc:1398] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.668752 14001 tablet_bootstrap.cc:654] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.668947 13998 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:51.670019 14001 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:51.670167 14001 ts_tablet_manager.cc:1367] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:58:51.670701 14001 raft_consensus.cc:348] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.670886 14001 raft_consensus.cc:374] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.670970 14001 raft_consensus.cc:729] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.671212 14001 consensus_queue.cc:260] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.672333 14001 ts_tablet_manager.cc:1398] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:51.673396 13998 tablet_bootstrap.cc:654] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.674506 13998 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:51.674583 13998 ts_tablet_manager.cc:1367] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:51.674966 13998 raft_consensus.cc:348] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.675088 13998 raft_consensus.cc:374] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.675117 13998 raft_consensus.cc:729] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.675243 13998 consensus_queue.cc:260] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.675516 13998 ts_tablet_manager.cc:1398] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:51.677125 13997 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:51.677261 13997 ts_tablet_manager.cc:1367] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 07:58:51.678170 13997 raft_consensus.cc:348] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.678653 13997 raft_consensus.cc:374] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.678748 13997 raft_consensus.cc:729] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.678843 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 4a02367c4c7a4e229af38dc27571fec1 (table rebalancer_dynamic_table_new_name [id=22ac1fc0008d462e9520946b78c8462c]) successfully deleted
I20240501 07:58:51.679049 13997 consensus_queue.cc:260] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.679472 14000 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:51.679580 14000 ts_tablet_manager.cc:1367] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20240501 07:58:51.680158 14000 raft_consensus.cc:348] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.680284 14000 raft_consensus.cc:374] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.680320 14000 raft_consensus.cc:729] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.680465 14000 consensus_queue.cc:260] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.680471 13997 ts_tablet_manager.cc:1398] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:51.680812 13997 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:51.680863 14000 ts_tablet_manager.cc:1398] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.681844 13999 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:51.681941 13999 ts_tablet_manager.cc:1367] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 07:58:51.682431 13999 raft_consensus.cc:348] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.682534 13999 raft_consensus.cc:374] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.682567 13999 raft_consensus.cc:729] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.682690 13999 consensus_queue.cc:260] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.682969 13999 ts_tablet_manager.cc:1398] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.683157 13999 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:51.683903 13997 tablet_bootstrap.cc:654] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.684012 13999 tablet_bootstrap.cc:654] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.684706 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:51.687812 13997 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:51.687891 13997 ts_tablet_manager.cc:1367] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:51.688421 13997 raft_consensus.cc:348] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.688588 13997 raft_consensus.cc:374] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.688666 13997 raft_consensus.cc:729] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.688843 13997 consensus_queue.cc:260] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.689285 13997 ts_tablet_manager.cc:1398] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.689704 13999 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:51.689806 13999 ts_tablet_manager.cc:1367] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:51.690178 13999 raft_consensus.cc:348] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.690274 13999 raft_consensus.cc:374] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.690307 13999 raft_consensus.cc:729] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.690428 13999 consensus_queue.cc:260] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.690685 13999 ts_tablet_manager.cc:1398] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.693012 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd69a4008664024a9bfa072fa55db3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9]), partition=
I20240501 07:58:51.693147 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd69a4008664024a9bfa072fa55db3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9]), partition=
I20240501 07:58:51.693298 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd69a4008664024a9bfa072fa55db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.693399 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd69a4008664024a9bfa072fa55db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.693612 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd69a4008664024a9bfa072fa55db3f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9]), partition=
I20240501 07:58:51.693840 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd69a4008664024a9bfa072fa55db3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.696820 14015 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:51.697883 14015 tablet_bootstrap.cc:654] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.699859 14015 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:51.699951 14015 ts_tablet_manager.cc:1367] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:51.700330 14015 raft_consensus.cc:348] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.700420 14015 raft_consensus.cc:374] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.700467 14015 raft_consensus.cc:729] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.700631 14015 consensus_queue.cc:260] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.700989 14015 ts_tablet_manager.cc:1398] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.704221 13806 raft_consensus.cc:482] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.704332 13806 raft_consensus.cc:504] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.704828 13806 leader_election.cc:288] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:51.705555 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a550051551640dcab9733948bb0966f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:51.705720 9534 raft_consensus.cc:2454] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:51.706036 9207 leader_election.cc:302] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.706269 13806 raft_consensus.cc:2781] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.706328 13806 raft_consensus.cc:482] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.706379 13806 raft_consensus.cc:3037] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.708068 13806 raft_consensus.cc:504] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.708539 13806 leader_election.cc:288] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:51.709465 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a550051551640dcab9733948bb0966f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:51.709610 9534 raft_consensus.cc:3037] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.710038 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a550051551640dcab9733948bb0966f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:51.710189 7010 raft_consensus.cc:3037] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.710827 9534 raft_consensus.cc:2454] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:51.711187 9207 leader_election.cc:302] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.711540 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a550051551640dcab9733948bb0966f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:51.711829 7010 raft_consensus.cc:2454] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:51.718144 14024 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:51.718549 13806 raft_consensus.cc:2781] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.718953 13806 raft_consensus.cc:686] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:51.719184 13806 consensus_queue.cc:237] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.721750 14024 tablet_bootstrap.cc:654] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.722339 6730 catalog_manager.cc:5484] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:51.725620 14024 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:51.725788 14024 ts_tablet_manager.cc:1367] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:51.726420 14024 raft_consensus.cc:348] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.726522 14024 raft_consensus.cc:374] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.726559 14024 raft_consensus.cc:729] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.726711 14024 consensus_queue.cc:260] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.727092 14024 ts_tablet_manager.cc:1398] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.727466 14001 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:51.732026 14001 tablet_bootstrap.cc:654] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:51.733832 9534 consensus_queue.cc:237] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:51.735632 7138 raft_consensus.cc:1261] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:51.735766 7010 raft_consensus.cc:1261] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:51.736306 11763 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:51.736722 11763 consensus_queue.cc:1035] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:51.738363 14001 tablet_bootstrap.cc:492] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:51.738471 14001 ts_tablet_manager.cc:1367] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 07:58:51.739104 14001 raft_consensus.cc:348] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.739243 14001 raft_consensus.cc:374] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:51.739282 14001 raft_consensus.cc:729] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:51.739431 14001 consensus_queue.cc:260] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.739816 14001 ts_tablet_manager.cc:1398] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:51.742529 11763 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:51.746820 6732 catalog_manager.cc:5484] T 0a81efefaf414a6091da247b0986c322 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:51.748596 7010 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:51.750185 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a81efefaf414a6091da247b0986c322 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:51.750401 13996 tablet_replica.cc:330] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:51.750543 13996 raft_consensus.cc:2229] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:51.750655 13996 raft_consensus.cc:2258] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:51.751143 13996 ts_tablet_manager.cc:1875] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:51.753038 7138 raft_consensus.cc:2932] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:51.753427 13996 ts_tablet_manager.cc:1888] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:51.753497 13996 log.cc:1196] T 0a81efefaf414a6091da247b0986c322 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0a81efefaf414a6091da247b0986c322
I20240501 07:58:51.754091 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 0a81efefaf414a6091da247b0986c322 (table rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]) successfully deleted
I20240501 07:58:51.823035 13301 raft_consensus.cc:482] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.823179 13301 raft_consensus.cc:504] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.823638 13301 leader_election.cc:288] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:51.824294 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b683d8146a34fc6b74315f7db5d6890" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:51.824455 9271 raft_consensus.cc:2454] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:51.824774 7207 leader_election.cc:302] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:51.824865 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b683d8146a34fc6b74315f7db5d6890" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:51.824954 13301 raft_consensus.cc:2781] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.825001 13301 raft_consensus.cc:482] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.825012 9534 raft_consensus.cc:2454] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:51.825043 13301 raft_consensus.cc:3037] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.826259 13301 raft_consensus.cc:504] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.826642 13301 leader_election.cc:288] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:51.827477 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b683d8146a34fc6b74315f7db5d6890" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:51.827597 9534 raft_consensus.cc:3037] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.828687 9534 raft_consensus.cc:2454] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:51.829123 7208 leader_election.cc:302] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:51.829408 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b683d8146a34fc6b74315f7db5d6890" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:51.829533 9271 raft_consensus.cc:3037] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.830545 9271 raft_consensus.cc:2454] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:51.831120 13301 raft_consensus.cc:2781] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.831233 13301 raft_consensus.cc:686] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:51.831404 13301 consensus_queue.cc:237] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:51.838795 6736 catalog_manager.cc:5484] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:51.851096 13314 raft_consensus.cc:482] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.851259 13314 raft_consensus.cc:504] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.851930 13314 leader_election.cc:288] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:51.853698 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd69a4008664024a9bfa072fa55db3f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:51.853865 9401 raft_consensus.cc:2454] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:51.855589 6814 leader_election.cc:302] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:51.855863 13314 raft_consensus.cc:2781] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.855933 13314 raft_consensus.cc:482] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.855988 13314 raft_consensus.cc:3037] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.857401 13314 raft_consensus.cc:504] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.858702 13314 leader_election.cc:288] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:51.859138 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd69a4008664024a9bfa072fa55db3f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:51.859270 9401 raft_consensus.cc:3037] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.860399 9401 raft_consensus.cc:2454] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:51.861135 6814 leader_election.cc:302] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:51.861320 13314 raft_consensus.cc:2781] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.861404 13314 raft_consensus.cc:686] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:51.861560 13314 consensus_queue.cc:237] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.865309 6736 catalog_manager.cc:5484] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:51.874764 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd69a4008664024a9bfa072fa55db3f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:51.874953 9666 raft_consensus.cc:2454] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:51.875262 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd69a4008664024a9bfa072fa55db3f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:51.875402 9665 raft_consensus.cc:3037] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.876556 9665 raft_consensus.cc:2454] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:51.922956 14080 raft_consensus.cc:482] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:51.923113 14080 raft_consensus.cc:504] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.923637 14080 leader_election.cc:288] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:51.923964 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a54e29c093e4be485bf1af1e9ea7a72" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:51.924052 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a54e29c093e4be485bf1af1e9ea7a72" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:51.924110 9271 raft_consensus.cc:2454] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:51.924202 6877 raft_consensus.cc:2454] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:51.924530 6945 leader_election.cc:302] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:51.924722 14080 raft_consensus.cc:2781] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:51.924782 14080 raft_consensus.cc:482] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:51.924816 14080 raft_consensus.cc:3037] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.926007 14080 raft_consensus.cc:504] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.926328 14080 leader_election.cc:288] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:51.926761 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a54e29c093e4be485bf1af1e9ea7a72" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:51.926888 6877 raft_consensus.cc:3037] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.928148 6877 raft_consensus.cc:2454] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:51.928473 6945 leader_election.cc:302] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:51.928683 14080 raft_consensus.cc:2781] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:51.928706 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a54e29c093e4be485bf1af1e9ea7a72" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:51.928840 14080 raft_consensus.cc:686] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:51.929034 14080 consensus_queue.cc:237] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:51.929195 9271 raft_consensus.cc:3037] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:51.931418 6733 catalog_manager.cc:5484] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:51.931695 9271 raft_consensus.cc:2454] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:51.953655 6859 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.957780 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:51.963387 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:51.964699 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.972285 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:51.981140 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:51.982267 9250 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:51.987076 9401 raft_consensus.cc:1261] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:51.987581 12885 consensus_queue.cc:1035] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.991158 14096 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:51.991612 14096 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:51.992764 6733 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 7fd69a4008664024a9bfa072fa55db3f (table rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9]). Expected version 1 got 0
I20240501 07:58:51.993891 9665 raft_consensus.cc:1261] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:51.994562 13314 consensus_queue.cc:1035] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:51.995201 9515 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:52.000308 6735 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 7fd69a4008664024a9bfa072fa55db3f (table rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9]). Expected version 1 got 0
I20240501 07:58:52.001112 14093 mvcc.cc:205] Tried to move back new op lower bound from 7022798159813292032 to 7022798159311626240. Current Snapshot: MvccSnapshot[applied={T|T < 7022798159813292032}]
I20240501 07:58:52.008131 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9] alter complete (version 1)
I20240501 07:58:52.010802 9383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.014150 14095 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:52.014767 14095 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:52.030097 14099 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:52.030462 14099 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:52.066332 9534 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:52.067574 9271 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:58:52.067631 9665 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:58:52.068490 11763 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.068711 11487 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.070484 14116 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:52.070986 9271 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:52.071038 9665 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:52.073571 6733 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:52.076251 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet ee29c58cdc6544e8933300ad5faecb3a with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:52.076804 14123 tablet_replica.cc:330] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:52.076961 14123 raft_consensus.cc:2229] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:52.077090 14123 raft_consensus.cc:2258] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.077607 14123 ts_tablet_manager.cc:1875] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:52.079728 14123 ts_tablet_manager.cc:1888] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 07:58:52.079792 14123 log.cc:1196] T ee29c58cdc6544e8933300ad5faecb3a P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee29c58cdc6544e8933300ad5faecb3a
I20240501 07:58:52.080327 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet ee29c58cdc6544e8933300ad5faecb3a (table rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]) successfully deleted
I20240501 07:58:52.099179 6877 consensus_queue.cc:237] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.100548 7270 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.100593 7138 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.100926 12885 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.101107 14141 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:52.101317 6815 consensus_peers.cc:485] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c0f4ea5ccba462d9468e3f445569eaa. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:52.102797 12885 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:52.103192 7270 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:52.106215 6734 catalog_manager.cc:5484] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:52.106457 7138 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:52.119804 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:52.125340 14095 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.125679 14095 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:52.125715 14096 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.126008 14096 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:52.126051 14099 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.126366 14099 tablet.cc:1620] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:52.129683 13806 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.131798 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=f55d88103b69406b83471019e82aecb9] alter complete (version 2)
I20240501 07:58:52.134153 12608 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.137603 6877 consensus_queue.cc:237] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.139115 7138 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.139127 7270 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.139595 12885 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.139755 14144 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.141855 12885 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:52.142305 7138 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
W20240501 07:58:52.144029 6814 consensus_peers.cc:485] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6421aafe6b34136bd8e1853cdf50786. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:52.144039 7270 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:52.146487 6735 catalog_manager.cc:5484] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:52.172426 7138 consensus_queue.cc:237] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.174005 7010 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.174132 7270 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:52.174481 12756 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.174849 14185 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:52.177793 14186 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:52.178269 7270 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
W20240501 07:58:52.180310 7076 consensus_peers.cc:485] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d9379f9bf86403ebf1695263db9a603. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:52.180960 7010 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:52.181834 6730 catalog_manager.cc:5484] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:52.235778 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:52.242450 14095 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.242810 14099 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.242990 14096 tablet.cc:1722] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.243428 13737 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.249428 13301 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.251298 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=f55d88103b69406b83471019e82aecb9] alter complete (version 3)
I20240501 07:58:52.291133 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:52.294178 6859 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.296875 9250 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:52.300995 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.308475 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:52.313980 9515 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:52.337051 9383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.351188 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.354168 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:52.354314 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:52.356607 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 7fd69a4008664024a9bfa072fa55db3f
I20240501 07:58:52.361655 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd69a4008664024a9bfa072fa55db3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:52 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:52.361990 14261 tablet_replica.cc:330] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:52.362214 14261 raft_consensus.cc:2229] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:52.362293 9383 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd69a4008664024a9bfa072fa55db3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:52 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:52.362476 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd69a4008664024a9bfa072fa55db3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:52 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:52.363940 14262 tablet_replica.cc:330] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:52.363997 14261 raft_consensus.cc:2258] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.364109 14262 raft_consensus.cc:2229] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:52.364254 14262 raft_consensus.cc:2258] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.364842 14262 ts_tablet_manager.cc:1875] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:52.364886 14261 ts_tablet_manager.cc:1875] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:52.367205 14262 ts_tablet_manager.cc:1888] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:52.367345 14262 log.cc:1196] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7fd69a4008664024a9bfa072fa55db3f
I20240501 07:58:52.367483 14261 ts_tablet_manager.cc:1888] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:52.367626 14261 log.cc:1196] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/7fd69a4008664024a9bfa072fa55db3f
I20240501 07:58:52.367709 14262 ts_tablet_manager.cc:1909] T 7fd69a4008664024a9bfa072fa55db3f P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:52.367962 14261 ts_tablet_manager.cc:1909] T 7fd69a4008664024a9bfa072fa55db3f P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:52.368448 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 7fd69a4008664024a9bfa072fa55db3f (table rebalancer_dynamic_table_new_name [id=f55d88103b69406b83471019e82aecb9]) successfully deleted
I20240501 07:58:52.368656 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 7fd69a4008664024a9bfa072fa55db3f (table rebalancer_dynamic_table_new_name [id=f55d88103b69406b83471019e82aecb9]) successfully deleted
I20240501 07:58:52.370294 14263 tablet_replica.cc:330] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:52.370499 14263 raft_consensus.cc:2229] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:52.375741 14263 raft_consensus.cc:2258] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.376423 14263 ts_tablet_manager.cc:1875] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:52.378708 14263 ts_tablet_manager.cc:1888] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:52.378783 14263 log.cc:1196] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7fd69a4008664024a9bfa072fa55db3f
I20240501 07:58:52.379141 14263 ts_tablet_manager.cc:1909] T 7fd69a4008664024a9bfa072fa55db3f P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:52.379880 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 7fd69a4008664024a9bfa072fa55db3f (table rebalancer_dynamic_table_new_name [id=f55d88103b69406b83471019e82aecb9]) successfully deleted
I20240501 07:58:52.387320 14080 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.394589 12648 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.454722 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:52.458159 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:52.463040 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d606ace12402a86ab5ce85a98cd44 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.463357 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d606ace12402a86ab5ce85a98cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.463619 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 80f08fbb906443759009170c6b5ad068 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.463703 9383 tablet_service.cc:1467] Processing CreateTablet for tablet 80f08fbb906443759009170c6b5ad068 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.463821 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f08fbb906443759009170c6b5ad068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.463958 9383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f08fbb906443759009170c6b5ad068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.465502 14303 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:52.465682 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d606ace12402a86ab5ce85a98cd44 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.465941 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d606ace12402a86ab5ce85a98cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.466393 9381 tablet_service.cc:1467] Processing CreateTablet for tablet f7c14004fdb24e96bacbf8589518f3b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.466646 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c14004fdb24e96bacbf8589518f3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.466723 14303 tablet_bootstrap.cc:654] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.467537 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d606ace12402a86ab5ce85a98cd44 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.467537 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f7c14004fdb24e96bacbf8589518f3b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.468010 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c14004fdb24e96bacbf8589518f3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.468170 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d606ace12402a86ab5ce85a98cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.470546 9250 tablet_service.cc:1467] Processing CreateTablet for tablet f7c14004fdb24e96bacbf8589518f3b1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.470796 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c14004fdb24e96bacbf8589518f3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.472903 14307 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:52.473897 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 80f08fbb906443759009170c6b5ad068 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:52.474187 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f08fbb906443759009170c6b5ad068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.474717 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]), partition=
I20240501 07:58:52.474908 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88e3e21ed84e53b5a9557d5b142799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.475878 14307 tablet_bootstrap.cc:654] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.476660 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]), partition=
I20240501 07:58:52.476887 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88e3e21ed84e53b5a9557d5b142799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.477051 14308 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:52.478075 14308 tablet_bootstrap.cc:654] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.479652 14307 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:52.479743 14307 ts_tablet_manager.cc:1367] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:52.479929 14306 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:52.480515 14307 raft_consensus.cc:348] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.480702 14307 raft_consensus.cc:374] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.480762 14307 raft_consensus.cc:729] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.480836 14306 tablet_bootstrap.cc:654] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.480988 14307 consensus_queue.cc:260] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.481420 14307 ts_tablet_manager.cc:1398] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:52.482172 14307 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:52.482439 14303 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:52.482547 14303 ts_tablet_manager.cc:1367] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:58:52.483119 14303 raft_consensus.cc:348] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.483219 14307 tablet_bootstrap.cc:654] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.483239 14303 raft_consensus.cc:374] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.483270 14303 raft_consensus.cc:729] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.483395 14303 consensus_queue.cc:260] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.483700 14303 ts_tablet_manager.cc:1398] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.484014 14303 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:52.483999 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]), partition=
I20240501 07:58:52.484261 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88e3e21ed84e53b5a9557d5b142799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.484452 14306 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:52.484726 14306 ts_tablet_manager.cc:1367] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:52.484958 14303 tablet_bootstrap.cc:654] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.485136 14306 raft_consensus.cc:348] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.485239 14306 raft_consensus.cc:374] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.485270 14306 raft_consensus.cc:729] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.485419 14306 consensus_queue.cc:260] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.485963 14306 ts_tablet_manager.cc:1398] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.486369 14306 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:52.487363 14306 tablet_bootstrap.cc:654] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.489944 14313 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:52.490408 14307 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:52.490558 14307 ts_tablet_manager.cc:1367] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 07:58:52.490936 14304 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:52.492389 14307 raft_consensus.cc:348] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.492569 14307 raft_consensus.cc:374] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.492653 14307 raft_consensus.cc:729] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.492863 14307 consensus_queue.cc:260] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.493398 14307 ts_tablet_manager.cc:1398] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:52.494552 14304 tablet_bootstrap.cc:654] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.494629 14313 tablet_bootstrap.cc:654] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.496357 14308 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:52.496501 14308 ts_tablet_manager.cc:1367] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:58:52.496707 14304 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:52.496784 14304 ts_tablet_manager.cc:1367] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:52.497061 14308 raft_consensus.cc:348] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.497179 14308 raft_consensus.cc:374] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.497217 14308 raft_consensus.cc:729] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.497267 14304 raft_consensus.cc:348] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.497361 14308 consensus_queue.cc:260] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.497401 14304 raft_consensus.cc:374] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.497586 14304 raft_consensus.cc:729] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.497721 14308 ts_tablet_manager.cc:1398] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.497870 14304 consensus_queue.cc:260] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.498657 14303 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:52.498740 14303 ts_tablet_manager.cc:1367] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:52.498927 14304 ts_tablet_manager.cc:1398] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:52.499238 14303 raft_consensus.cc:348] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.499356 14303 raft_consensus.cc:374] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.499390 14303 raft_consensus.cc:729] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.499516 14304 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:52.499516 14303 consensus_queue.cc:260] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.499814 14303 ts_tablet_manager.cc:1398] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.501627 14304 tablet_bootstrap.cc:654] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.503600 14304 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:52.503685 14304 ts_tablet_manager.cc:1367] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:52.504174 14304 raft_consensus.cc:348] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.504288 14304 raft_consensus.cc:374] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.504323 14304 raft_consensus.cc:729] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.504442 14304 consensus_queue.cc:260] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.504765 14304 ts_tablet_manager.cc:1398] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.504925 14304 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:52.505704 14304 tablet_bootstrap.cc:654] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.510835 14306 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:52.510951 14306 ts_tablet_manager.cc:1367] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 07:58:52.511698 14306 raft_consensus.cc:348] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.511845 14306 raft_consensus.cc:374] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.511880 14306 raft_consensus.cc:729] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.512015 14306 consensus_queue.cc:260] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.514181 14306 ts_tablet_manager.cc:1398] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:52.514909 14306 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:52.518621 14306 tablet_bootstrap.cc:654] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:52.520859 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:52.521205 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:52.525300 9507 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:52.533423 14313 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:52.533655 14313 ts_tablet_manager.cc:1367] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20240501 07:58:52.534193 14313 raft_consensus.cc:348] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.534909 14313 raft_consensus.cc:374] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.535034 14313 raft_consensus.cc:729] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.535245 14313 consensus_queue.cc:260] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.535635 14313 ts_tablet_manager.cc:1398] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:52.535916 9250 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:52.537235 14306 tablet_bootstrap.cc:492] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:52.544369 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.546834 14116 raft_consensus.cc:482] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:52.546937 14116 raft_consensus.cc:504] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.547322 14116 leader_election.cc:288] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:52.547722 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:52.547856 7140 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:52.547928 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:52.548092 9402 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:52.548441 9469 leader_election.cc:302] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:52.548982 14116 raft_consensus.cc:2781] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:52.549043 14116 raft_consensus.cc:482] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:52.549078 14116 raft_consensus.cc:3037] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.550112 14116 raft_consensus.cc:504] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.550410 14116 leader_election.cc:288] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:52.550674 12608 raft_consensus.cc:482] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:52.550819 12608 raft_consensus.cc:504] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.551172 12608 leader_election.cc:288] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:52.551131 14306 ts_tablet_manager.cc:1367] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.001s
I20240501 07:58:52.551908 14306 raft_consensus.cc:348] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.551975 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:52.552074 7140 raft_consensus.cc:3037] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.552145 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:52.552253 9402 raft_consensus.cc:3037] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.553031 7140 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:52.553279 9402 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:52.553560 14306 raft_consensus.cc:374] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.553630 9469 leader_election.cc:302] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:52.553674 14306 raft_consensus.cc:729] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.553840 11763 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:52.553879 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c88e3e21ed84e53b5a9557d5b142799" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:52.553921 11763 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.553939 14306 consensus_queue.cc:260] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.554067 9665 raft_consensus.cc:2454] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:52.554220 11763 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:52.554304 11763 raft_consensus.cc:2781] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:52.554334 14306 ts_tablet_manager.cc:1398] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:52.554369 11763 raft_consensus.cc:686] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:52.554384 9206 leader_election.cc:302] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:52.554491 11763 consensus_queue.cc:237] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.554641 12608 raft_consensus.cc:2781] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:52.554687 12608 raft_consensus.cc:482] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:52.554731 12608 raft_consensus.cc:3037] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.555801 12608 raft_consensus.cc:504] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.556145 12608 leader_election.cc:288] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:52.556890 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c88e3e21ed84e53b5a9557d5b142799" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:52.556991 9666 raft_consensus.cc:3037] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.557762 6735 catalog_manager.cc:5484] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:52.557881 9666 raft_consensus.cc:2454] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:52.558140 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.559176 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:52.559288 9665 raft_consensus.cc:2454] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:52.559405 9206 leader_election.cc:302] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:52.559798 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:52.560161 12608 raft_consensus.cc:2781] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:52.560138 9470 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
W20240501 07:58:52.560318 9470 leader_election.cc:341] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:52.560565 14116 raft_consensus.cc:2781] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:52.560612 14116 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:52.560647 14116 raft_consensus.cc:3037] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.560765 12608 raft_consensus.cc:686] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:52.560911 12608 consensus_queue.cc:237] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.562784 14116 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.563170 14116 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:52.563488 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:52.563606 9665 raft_consensus.cc:3037] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.563592 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:52.563658 6735 catalog_manager.cc:5484] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
W20240501 07:58:52.564040 9470 leader_election.cc:341] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:52.564575 9665 raft_consensus.cc:2454] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:52.569131 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c88e3e21ed84e53b5a9557d5b142799" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:52.569289 7271 raft_consensus.cc:2454] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:52.569463 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c88e3e21ed84e53b5a9557d5b142799" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:52.569553 7272 raft_consensus.cc:3037] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.570075 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.570556 7272 raft_consensus.cc:2454] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:52.571350 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:52.572386 9470 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters: fa6964452854415a85ad72f8639768d6
I20240501 07:58:52.573212 14116 raft_consensus.cc:2781] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:52.573979 14116 raft_consensus.cc:686] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:52.574172 14116 consensus_queue.cc:237] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.576172 6735 catalog_manager.cc:5484] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:52.584153 14335 ts_tablet_manager.cc:902] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:52.584766 14335 tablet_copy_client.cc:308] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:52.585453 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 8c0f4ea5ccba462d9468e3f445569eaa from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:34317)
I20240501 07:58:52.585578 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 8c0f4ea5ccba462d9468e3f445569eaa from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:34317: session id = ac6c8bc1b74b483791e262abee7862cf-8c0f4ea5ccba462d9468e3f445569eaa
I20240501 07:58:52.586539 6899 tablet_copy_source_session.cc:205] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:52.587247 14304 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:52.587332 14304 ts_tablet_manager.cc:1367] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.000s
I20240501 07:58:52.587759 14304 raft_consensus.cc:348] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.587864 14304 raft_consensus.cc:374] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:52.587891 14304 raft_consensus.cc:729] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.587981 14304 consensus_queue.cc:260] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.588227 14304 ts_tablet_manager.cc:1398] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.591367 14335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0f4ea5ccba462d9468e3f445569eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.594653 14335 tablet_copy_client.cc:783] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:52.594913 14335 tablet_copy_client.cc:647] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:52.596028 14335 tablet_copy_client.cc:523] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.597450 14335 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:52.611263 14334 ts_tablet_manager.cc:902] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:52.611871 14334 tablet_copy_client.cc:308] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:58:52.612675 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet b6421aafe6b34136bd8e1853cdf50786 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:44257)
I20240501 07:58:52.612771 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet b6421aafe6b34136bd8e1853cdf50786 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:44257: session id = 7fa3a88c25964445975bb47e0feed460-b6421aafe6b34136bd8e1853cdf50786
I20240501 07:58:52.613698 6899 tablet_copy_source_session.cc:205] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:52.615072 14334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6421aafe6b34136bd8e1853cdf50786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.617615 14334 tablet_copy_client.cc:783] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:52.617797 14334 tablet_copy_client.cc:647] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:52.624089 14334 tablet_copy_client.cc:523] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.626174 14334 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:52.630553 14351 ts_tablet_manager.cc:902] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:52.635687 14351 tablet_copy_client.cc:308] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:52.636057 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 2d9379f9bf86403ebf1695263db9a603 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:60769)
I20240501 07:58:52.636155 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 2d9379f9bf86403ebf1695263db9a603 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:60769: session id = fa6964452854415a85ad72f8639768d6-2d9379f9bf86403ebf1695263db9a603
I20240501 07:58:52.637043 7160 tablet_copy_source_session.cc:205] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:52.637403 12608 raft_consensus.cc:482] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:52.637516 12608 raft_consensus.cc:504] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.637893 14351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9379f9bf86403ebf1695263db9a603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:52.637888 12608 leader_election.cc:288] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:52.638411 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c14004fdb24e96bacbf8589518f3b1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:52.638520 9402 raft_consensus.cc:2454] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:52.638720 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c14004fdb24e96bacbf8589518f3b1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:52.638828 9665 raft_consensus.cc:2454] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:52.639294 14335 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.639335 9206 leader_election.cc:302] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:52.639575 14335 tablet_bootstrap.cc:492] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:58:52.639737 14335 ts_tablet_manager.cc:1367] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.042s user 0.025s sys 0.000s
I20240501 07:58:52.639803 12608 raft_consensus.cc:2781] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:52.639860 12608 raft_consensus.cc:482] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:52.639914 12608 raft_consensus.cc:3037] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.640254 14335 raft_consensus.cc:348] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.640347 14335 raft_consensus.cc:729] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.640457 14335 consensus_queue.cc:260] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.640763 14335 ts_tablet_manager.cc:1398] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:52.640933 12608 raft_consensus.cc:504] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.641219 12608 leader_election.cc:288] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:52.641554 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c14004fdb24e96bacbf8589518f3b1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:52.641654 9402 raft_consensus.cc:3037] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.641683 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c14004fdb24e96bacbf8589518f3b1" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:52.641772 9665 raft_consensus.cc:3037] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.642554 9402 raft_consensus.cc:2454] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:52.642606 9665 raft_consensus.cc:2454] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:52.642753 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-8c0f4ea5ccba462d9468e3f445569eaa received from {username='slave'} at 127.0.99.135:34317
I20240501 07:58:52.642835 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-8c0f4ea5ccba462d9468e3f445569eaa on tablet 8c0f4ea5ccba462d9468e3f445569eaa with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:52.643023 9206 leader_election.cc:302] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:52.643224 12608 raft_consensus.cc:2781] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:52.643297 12608 raft_consensus.cc:686] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:52.643424 14351 tablet_copy_client.cc:783] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:52.643406 12608 consensus_queue.cc:237] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.643559 14351 tablet_copy_client.cc:647] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:52.645165 14351 tablet_copy_client.cc:523] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.646348 6732 catalog_manager.cc:5484] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:52.646653 14351 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:52.665033 14334 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.665333 14334 tablet_bootstrap.cc:492] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:52.665503 14334 ts_tablet_manager.cc:1367] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.000s
I20240501 07:58:52.665944 14334 raft_consensus.cc:348] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.666076 14334 raft_consensus.cc:729] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.666234 14334 consensus_queue.cc:260] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:52.667384 14334 ts_tablet_manager.cc:1398] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:52.668541 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-b6421aafe6b34136bd8e1853cdf50786 received from {username='slave'} at 127.0.99.136:44257
I20240501 07:58:52.668628 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-b6421aafe6b34136bd8e1853cdf50786 on tablet b6421aafe6b34136bd8e1853cdf50786 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:52.672338 14351 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.672646 14351 tablet_bootstrap.cc:492] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:52.672813 14351 ts_tablet_manager.cc:1367] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.004s
I20240501 07:58:52.673434 14351 raft_consensus.cc:348] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.673559 14351 raft_consensus.cc:729] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:52.673614 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:52.673705 14351 consensus_queue.cc:260] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:52.675308 14351 ts_tablet_manager.cc:1398] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:52.675829 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-2d9379f9bf86403ebf1695263db9a603 received from {username='slave'} at 127.0.99.134:60769
I20240501 07:58:52.675918 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session fa6964452854415a85ad72f8639768d6-2d9379f9bf86403ebf1695263db9a603 on tablet 2d9379f9bf86403ebf1695263db9a603 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:52.678198 7272 raft_consensus.cc:1261] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:52.679437 9665 raft_consensus.cc:1261] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:52.679914 12608 consensus_queue.cc:1035] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.680054 13806 consensus_queue.cc:1035] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:52.683161 6732 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]). Expected version 1 got 0
I20240501 07:58:52.683743 14233 mvcc.cc:205] Tried to move back new op lower bound from 7022798162646097920 to 7022798162171543552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798162646097920}]
I20240501 07:58:52.683876 14099 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:52.684168 14099 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:52.684454 14367 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:52.684768 14367 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:52.686203 6732 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]). Expected version 1 got 0
I20240501 07:58:52.687702 6735 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641]). Expected version 1 got 0
I20240501 07:58:52.688542 14368 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:52.688897 14368 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:52.690063 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641] alter complete (version 1)
I20240501 07:58:52.762413 9250 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:52.769316 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:52.769757 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:52.771255 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:52.781478 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.786886 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.790607 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:52.795559 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:52.803406 14367 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.803730 14367 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:52.804992 14099 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.805296 14099 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:52.806259 14368 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:52.806561 14368 tablet.cc:1620] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:52.812846 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e59fc8085f6b4986be32491ab43f8641] alter complete (version 2)
I20240501 07:58:52.830245 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.873481 9534 consensus_queue.cc:237] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:52.874619 7140 raft_consensus.cc:1261] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:52.874662 9665 raft_consensus.cc:1261] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:52.875077 14116 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:52.875269 11487 consensus_queue.cc:1035] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:52.877272 11487 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:52.877846 9665 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:52.880458 6736 catalog_manager.cc:5484] T 1037ad1dda6448f7a026fc685642d5c6 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:52.880633 7140 raft_consensus.cc:2932] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:52.883785 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037ad1dda6448f7a026fc685642d5c6 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:52.884492 14433 tablet_replica.cc:330] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:52.884645 14433 raft_consensus.cc:2229] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:52.884768 14433 raft_consensus.cc:2258] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.885151 14433 ts_tablet_manager.cc:1875] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:52.887451 14433 ts_tablet_manager.cc:1888] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:52.887567 14433 log.cc:1196] T 1037ad1dda6448f7a026fc685642d5c6 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1037ad1dda6448f7a026fc685642d5c6
I20240501 07:58:52.888199 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 1037ad1dda6448f7a026fc685642d5c6 (table rebalancer_extra_table_6 [id=ce32719f30ae4bbe8111f87256d25cbe]) successfully deleted
I20240501 07:58:52.915102 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:52.918952 12912 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:52.919078 12912 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:52.919512 12912 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:52.919809 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:52.919991 9666 raft_consensus.cc:2379] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa6964452854415a85ad72f8639768d6 in current term 1: Already voted for candidate ac6c8bc1b74b483791e262abee7862cf in this term.
I20240501 07:58:52.920313 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:52.920883 14367 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.921085 9339 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:52.921303 14099 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.925496 12912 raft_consensus.cc:3037] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:52.926551 12912 raft_consensus.cc:2726] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:52.928248 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=e59fc8085f6b4986be32491ab43f8641] alter complete (version 3)
I20240501 07:58:52.929316 14368 tablet.cc:1722] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:52.981914 9250 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:52.988956 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:52.989327 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:52.991313 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:52.995532 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.003756 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:53.011940 14116 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.013145 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.019755 14116 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.029901 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:53.032287 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:53.032429 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:53.034595 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 8c88e3e21ed84e53b5a9557d5b142799
I20240501 07:58:53.035149 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:53.035326 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:53.035477 14485 tablet_replica.cc:330] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:53.035645 14485 raft_consensus.cc:2229] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.035790 14485 raft_consensus.cc:2258] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.036384 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c88e3e21ed84e53b5a9557d5b142799 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:53.036361 11487 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.036434 14485 ts_tablet_manager.cc:1875] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.038784 14485 ts_tablet_manager.cc:1888] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.038900 14485 log.cc:1196] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8c88e3e21ed84e53b5a9557d5b142799
I20240501 07:58:53.039265 14485 ts_tablet_manager.cc:1909] T 8c88e3e21ed84e53b5a9557d5b142799 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:53.039934 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table_new_name [id=e59fc8085f6b4986be32491ab43f8641]) successfully deleted
I20240501 07:58:53.048451 14488 tablet_replica.cc:330] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:53.048734 14488 raft_consensus.cc:2229] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:53.049423 14488 raft_consensus.cc:2258] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.050094 14488 ts_tablet_manager.cc:1875] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.051874 11763 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.052444 14488 ts_tablet_manager.cc:1888] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.052574 14488 log.cc:1196] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8c88e3e21ed84e53b5a9557d5b142799
I20240501 07:58:53.052846 14488 ts_tablet_manager.cc:1909] T 8c88e3e21ed84e53b5a9557d5b142799 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:53.053593 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table_new_name [id=e59fc8085f6b4986be32491ab43f8641]) successfully deleted
I20240501 07:58:53.055090 14486 tablet_replica.cc:330] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:53.055238 14486 raft_consensus.cc:2229] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.055358 14486 raft_consensus.cc:2258] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.059415 14486 ts_tablet_manager.cc:1875] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.067858 14486 ts_tablet_manager.cc:1888] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.067934 14486 log.cc:1196] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8c88e3e21ed84e53b5a9557d5b142799
I20240501 07:58:53.068177 14486 ts_tablet_manager.cc:1909] T 8c88e3e21ed84e53b5a9557d5b142799 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:53.071586 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 8c88e3e21ed84e53b5a9557d5b142799 (table rebalancer_dynamic_table_new_name [id=e59fc8085f6b4986be32491ab43f8641]) successfully deleted
I20240501 07:58:53.094859 13806 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.098189 12608 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.117285 9666 raft_consensus.cc:1203] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:53.137094 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:53.137786 9534 raft_consensus.cc:1203] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:53.146150 9402 raft_consensus.cc:1203] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:53.149506 9250 tablet_service.cc:1467] Processing CreateTablet for tablet f46a78e76be24ababa66b6f5eb92c696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]), partition=
I20240501 07:58:53.149636 9645 tablet_service.cc:1467] Processing CreateTablet for tablet f46a78e76be24ababa66b6f5eb92c696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]), partition=
I20240501 07:58:53.149636 9507 tablet_service.cc:1467] Processing CreateTablet for tablet f46a78e76be24ababa66b6f5eb92c696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]), partition=
I20240501 07:58:53.149791 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46a78e76be24ababa66b6f5eb92c696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.149873 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46a78e76be24ababa66b6f5eb92c696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.149870 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46a78e76be24ababa66b6f5eb92c696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.152936 14534 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:53.154525 14533 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:53.155041 14534 tablet_bootstrap.cc:654] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.155676 14533 tablet_bootstrap.cc:654] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.155716 14535 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:53.157179 14533 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:53.157742 14533 ts_tablet_manager.cc:1367] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:53.158622 14533 raft_consensus.cc:348] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.158828 14533 raft_consensus.cc:374] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.158914 14533 raft_consensus.cc:729] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.159080 14533 consensus_queue.cc:260] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.160372 14533 ts_tablet_manager.cc:1398] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:53.160810 14534 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:53.160926 14534 ts_tablet_manager.cc:1367] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 07:58:53.162765 14534 raft_consensus.cc:348] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.162869 14534 raft_consensus.cc:374] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.162904 14534 raft_consensus.cc:729] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.163067 14534 consensus_queue.cc:260] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.166658 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:53.167155 14534 ts_tablet_manager.cc:1398] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:53.167562 12608 raft_consensus.cc:482] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.167655 12608 raft_consensus.cc:504] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.168009 12608 leader_election.cc:288] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.168476 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46a78e76be24ababa66b6f5eb92c696" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:53.168784 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46a78e76be24ababa66b6f5eb92c696" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
W20240501 07:58:53.168834 9207 leader_election.cc:341] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:53.168907 9666 raft_consensus.cc:2454] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:53.169838 9206 leader_election.cc:302] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:53.170056 12608 raft_consensus.cc:2781] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:53.170118 12608 raft_consensus.cc:482] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:53.170162 12608 raft_consensus.cc:3037] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.171218 12608 raft_consensus.cc:504] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.171558 12608 leader_election.cc:288] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.171936 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46a78e76be24ababa66b6f5eb92c696" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
W20240501 07:58:53.172279 9207 leader_election.cc:341] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:53.172380 14535 tablet_bootstrap.cc:654] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.172863 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46a78e76be24ababa66b6f5eb92c696" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:53.172968 9666 raft_consensus.cc:3037] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.173902 9666 raft_consensus.cc:2454] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:53.174850 9507 tablet_service.cc:1467] Processing CreateTablet for tablet da18394593a24e61ba2f90356645621a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.174994 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 8123a9666b814a8eb93098bf850ebd72 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.175130 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da18394593a24e61ba2f90356645621a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.175273 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8123a9666b814a8eb93098bf850ebd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.174991 7121 tablet_service.cc:1467] Processing CreateTablet for tablet da18394593a24e61ba2f90356645621a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.176019 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da18394593a24e61ba2f90356645621a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.175315 7120 tablet_service.cc:1467] Processing CreateTablet for tablet f4729eaaa94e480ea9bf075a355dffa7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.178586 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4729eaaa94e480ea9bf075a355dffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.179396 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 8123a9666b814a8eb93098bf850ebd72 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.179396 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 8123a9666b814a8eb93098bf850ebd72 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.179642 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8123a9666b814a8eb93098bf850ebd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.179666 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8123a9666b814a8eb93098bf850ebd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.181152 9645 tablet_service.cc:1467] Processing CreateTablet for tablet da18394593a24e61ba2f90356645621a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.181349 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da18394593a24e61ba2f90356645621a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.181753 14534 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:53.182121 14551 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:53.182605 14534 tablet_bootstrap.cc:654] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.183377 6989 tablet_service.cc:1467] Processing CreateTablet for tablet f4729eaaa94e480ea9bf075a355dffa7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.183653 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4729eaaa94e480ea9bf075a355dffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.185981 9250 tablet_service.cc:1467] Processing CreateTablet for tablet f4729eaaa94e480ea9bf075a355dffa7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:53.186214 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4729eaaa94e480ea9bf075a355dffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.187049 9206 leader_election.cc:302] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:53.187716 13806 raft_consensus.cc:2781] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:53.187796 13806 raft_consensus.cc:686] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:53.187927 14551 tablet_bootstrap.cc:654] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.187937 13806 consensus_queue.cc:237] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.188714 14553 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:53.189685 14553 tablet_bootstrap.cc:654] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.190866 14551 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:53.191335 14551 ts_tablet_manager.cc:1367] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:53.192835 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:53.193598 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:53.193594 14551 raft_consensus.cc:348] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.193843 14551 raft_consensus.cc:374] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.193921 14551 raft_consensus.cc:729] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.194159 14551 consensus_queue.cc:260] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.196156 14551 ts_tablet_manager.cc:1398] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:53.198055 14553 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:53.198210 14553 ts_tablet_manager.cc:1367] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:58:53.198710 14553 raft_consensus.cc:348] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.198936 14553 raft_consensus.cc:374] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.199047 14553 raft_consensus.cc:729] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.199294 14553 consensus_queue.cc:260] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.199748 14553 ts_tablet_manager.cc:1398] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:53.209218 14533 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:53.210083 6989 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.210857 9250 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:53.211194 14533 tablet_bootstrap.cc:654] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.213259 14535 tablet_bootstrap.cc:492] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:53.213347 14535 ts_tablet_manager.cc:1367] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.000s
I20240501 07:58:53.213742 14535 raft_consensus.cc:348] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.213845 14535 raft_consensus.cc:374] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.213871 14535 raft_consensus.cc:729] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.213981 14535 consensus_queue.cc:260] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.214267 14535 ts_tablet_manager.cc:1398] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:53.214282 14534 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:53.214365 14534 ts_tablet_manager.cc:1367] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.001s
I20240501 07:58:53.214440 14535 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:53.214808 14534 raft_consensus.cc:348] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.214920 14534 raft_consensus.cc:374] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.214948 14534 raft_consensus.cc:729] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.215112 14534 consensus_queue.cc:260] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.215405 14535 tablet_bootstrap.cc:654] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.215421 14534 ts_tablet_manager.cc:1398] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:53.215607 14534 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:53.216732 14534 tablet_bootstrap.cc:654] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.216898 14535 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:53.216969 14535 ts_tablet_manager.cc:1367] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:53.217365 14535 raft_consensus.cc:348] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.217480 14535 raft_consensus.cc:374] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.217535 14535 raft_consensus.cc:729] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.217628 14535 consensus_queue.cc:260] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.217885 14535 ts_tablet_manager.cc:1398] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:53.218108 14535 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:53.218607 14534 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:53.218746 14534 ts_tablet_manager.cc:1367] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:53.219239 14534 raft_consensus.cc:348] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.219390 14534 raft_consensus.cc:374] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.219484 14534 raft_consensus.cc:729] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.219686 14534 consensus_queue.cc:260] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.220028 14534 ts_tablet_manager.cc:1398] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:53.220099 14535 tablet_bootstrap.cc:654] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.222568 14535 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:53.222671 14535 ts_tablet_manager.cc:1367] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:53.223119 14535 raft_consensus.cc:348] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.223232 14535 raft_consensus.cc:374] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.223275 14535 raft_consensus.cc:729] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.223407 14535 consensus_queue.cc:260] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.223747 14535 ts_tablet_manager.cc:1398] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:53.225596 6729 catalog_manager.cc:5484] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:53.227602 14533 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:53.227710 14533 ts_tablet_manager.cc:1367] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 07:58:53.228173 14533 raft_consensus.cc:348] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.228302 14533 raft_consensus.cc:374] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.228331 14533 raft_consensus.cc:729] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.228446 14533 consensus_queue.cc:260] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.228772 14533 ts_tablet_manager.cc:1398] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:53.233755 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.240630 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:53.247593 9645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:53.248644 14550 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:53.252566 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.253793 14550 tablet_bootstrap.cc:654] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.264771 14550 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:53.264947 14550 ts_tablet_manager.cc:1367] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 07:58:53.265105 12648 raft_consensus.cc:482] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.265202 12648 raft_consensus.cc:504] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.265487 14550 raft_consensus.cc:348] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.265570 12648 leader_election.cc:288] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:53.265601 14550 raft_consensus.cc:374] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.265633 14550 raft_consensus.cc:729] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.265761 14550 consensus_queue.cc:260] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.266139 14550 ts_tablet_manager.cc:1398] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:53.266367 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:53.266501 9271 raft_consensus.cc:2454] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:53.266752 14550 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:53.266790 6945 leader_election.cc:302] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:53.266945 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:53.267235 12648 raft_consensus.cc:2781] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:53.267283 12648 raft_consensus.cc:482] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:53.267316 12648 raft_consensus.cc:3037] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.267829 14550 tablet_bootstrap.cc:654] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.268322 12648 raft_consensus.cc:504] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.268651 12648 leader_election.cc:288] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:53.269439 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
W20240501 07:58:53.269697 6944 leader_election.cc:341] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
W20240501 07:58:53.270395 6944 leader_election.cc:341] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:53.270925 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:53.271114 9271 raft_consensus.cc:3037] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.271854 14550 tablet_bootstrap.cc:492] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:53.271940 14550 ts_tablet_manager.cc:1367] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:53.272090 9271 raft_consensus.cc:2454] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:53.272334 6945 leader_election.cc:302] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters: fc946354b5d04503a56122e59fc320be
I20240501 07:58:53.272378 14550 raft_consensus.cc:348] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.272490 14550 raft_consensus.cc:374] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.272522 14550 raft_consensus.cc:729] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.272640 14550 consensus_queue.cc:260] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.272909 14550 ts_tablet_manager.cc:1398] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:53.273149 12648 raft_consensus.cc:2781] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:53.273226 12648 raft_consensus.cc:686] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:53.273353 12648 consensus_queue.cc:237] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.275691 6733 catalog_manager.cc:5484] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:53.297504 14532 raft_consensus.cc:482] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.297652 14532 raft_consensus.cc:504] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.298099 14532 leader_election.cc:288] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:53.298408 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:53.298662 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4729eaaa94e480ea9bf075a355dffa7" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:53.298840 9271 raft_consensus.cc:2379] T f4729eaaa94e480ea9bf075a355dffa7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc946354b5d04503a56122e59fc320be in current term 1: Already voted for candidate 9a3888360d4d4ace96b6b909c160bfc2 in this term.
I20240501 07:58:53.299384 7077 leader_election.cc:302] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:58:53.299921 14532 raft_consensus.cc:3037] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.301261 14532 raft_consensus.cc:2726] T f4729eaaa94e480ea9bf075a355dffa7 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:53.301668 13577 raft_consensus.cc:482] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.301765 13577 raft_consensus.cc:504] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.302141 13577 leader_election.cc:288] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.302440 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8123a9666b814a8eb93098bf850ebd72" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:53.302565 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8123a9666b814a8eb93098bf850ebd72" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:53.302597 9402 raft_consensus.cc:2454] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:53.302692 9534 raft_consensus.cc:2454] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:53.303066 9601 leader_election.cc:302] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:53.303284 13577 raft_consensus.cc:2781] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:53.303344 13577 raft_consensus.cc:482] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:53.303397 13577 raft_consensus.cc:3037] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.304246 13577 raft_consensus.cc:504] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.304569 13577 leader_election.cc:288] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.304884 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8123a9666b814a8eb93098bf850ebd72" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:53.304955 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8123a9666b814a8eb93098bf850ebd72" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:53.305001 9402 raft_consensus.cc:3037] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.305078 9534 raft_consensus.cc:3037] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.305732 9402 raft_consensus.cc:2454] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:53.306035 9601 leader_election.cc:302] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:53.306175 9534 raft_consensus.cc:2454] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:53.306521 13577 raft_consensus.cc:2781] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:53.306607 13577 raft_consensus.cc:686] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:53.306746 13577 consensus_queue.cc:237] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:53.310308 6728 catalog_manager.cc:5484] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:53.336807 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:53.343221 9534 raft_consensus.cc:3037] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.343322 9666 raft_consensus.cc:1261] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:53.343778 13806 consensus_queue.cc:1035] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.344326 9534 raft_consensus.cc:1261] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:53.345932 13806 consensus_queue.cc:1035] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.347230 6728 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]). Expected version 1 got 0
I20240501 07:58:53.347951 14367 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:53.348273 14367 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:53.349403 6728 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]). Expected version 1 got 0
I20240501 07:58:53.349444 14099 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:53.349721 14099 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:53.354660 14588 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:53.354868 6735 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8]). Expected version 1 got 0
I20240501 07:58:53.355252 14588 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:53.356233 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8] alter complete (version 1)
I20240501 07:58:53.459719 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:53.464939 14367 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:53.465278 14367 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:53.465312 14099 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:53.465620 14099 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:53.466106 14588 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:53.466388 14588 tablet.cc:1620] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:53.470144 13577 raft_consensus.cc:482] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.470253 13577 raft_consensus.cc:504] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.470701 13577 leader_election.cc:288] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.471038 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da18394593a24e61ba2f90356645621a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:53.471069 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da18394593a24e61ba2f90356645621a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:53.471168 9534 raft_consensus.cc:2454] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:53.471168 7140 raft_consensus.cc:2454] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:53.471431 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=1dd60d66aa1648d783009ecca1586da8] alter complete (version 2)
I20240501 07:58:53.471729 9602 leader_election.cc:302] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:53.471935 13577 raft_consensus.cc:2781] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:53.471992 13577 raft_consensus.cc:482] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:53.472028 13577 raft_consensus.cc:3037] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.472891 13577 raft_consensus.cc:504] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.473198 13577 leader_election.cc:288] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:53.473441 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da18394593a24e61ba2f90356645621a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:53.473441 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da18394593a24e61ba2f90356645621a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:53.473546 9534 raft_consensus.cc:3037] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.473556 7140 raft_consensus.cc:3037] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.474454 9534 raft_consensus.cc:2454] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:53.474454 7140 raft_consensus.cc:2454] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:53.474720 9602 leader_election.cc:302] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:53.474967 13577 raft_consensus.cc:2781] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:53.475085 13577 raft_consensus.cc:686] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:53.475225 13577 consensus_queue.cc:237] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.477150 6728 catalog_manager.cc:5484] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.507400 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:53.512008 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:53.513752 9250 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:53.525182 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.525646 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:53.526806 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.537389 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.540344 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:53.575662 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:53.582692 14367 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:53.583554 14099 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:53.584475 14588 tablet.cc:1722] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:53.589202 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=1dd60d66aa1648d783009ecca1586da8] alter complete (version 3)
I20240501 07:58:53.620862 7140 consensus_queue.cc:237] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:53.622146 9271 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:53.622211 6877 raft_consensus.cc:1261] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:53.622578 14532 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:53.622793 14450 consensus_queue.cc:1035] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:53.624874 14450 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:53.625483 9271 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:53.625919 6877 raft_consensus.cc:2932] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:53.628852 6737 catalog_manager.cc:5484] T 2eec88312feb44feb70401fe29a3bb83 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:53.632599 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eec88312feb44feb70401fe29a3bb83 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:53.632957 14646 tablet_replica.cc:330] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:53.633095 14646 raft_consensus.cc:2229] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.633239 14646 raft_consensus.cc:2258] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.633811 14646 ts_tablet_manager.cc:1875] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:53.636191 14646 ts_tablet_manager.cc:1888] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:53.636268 14646 log.cc:1196] T 2eec88312feb44feb70401fe29a3bb83 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2eec88312feb44feb70401fe29a3bb83
I20240501 07:58:53.636787 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 2eec88312feb44feb70401fe29a3bb83 (table rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]) successfully deleted
I20240501 07:58:53.692941 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:53.693099 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:53.698933 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet f46a78e76be24ababa66b6f5eb92c696
I20240501 07:58:53.703156 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet f46a78e76be24ababa66b6f5eb92c696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:53.703446 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet f46a78e76be24ababa66b6f5eb92c696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:53.703702 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet f46a78e76be24ababa66b6f5eb92c696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:53 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:53.704859 14684 tablet_replica.cc:330] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:53.709127 14684 raft_consensus.cc:2229] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:53.709607 14684 raft_consensus.cc:2258] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.710613 14684 ts_tablet_manager.cc:1875] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.715145 14683 tablet_replica.cc:330] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:53.715353 14685 tablet_replica.cc:330] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:53.716023 14684 ts_tablet_manager.cc:1888] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.716164 14684 log.cc:1196] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f46a78e76be24ababa66b6f5eb92c696
I20240501 07:58:53.716310 14683 raft_consensus.cc:2229] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.716425 14683 raft_consensus.cc:2258] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.716611 14685 raft_consensus.cc:2229] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.716756 14685 raft_consensus.cc:2258] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.716756 14684 ts_tablet_manager.cc:1909] T f46a78e76be24ababa66b6f5eb92c696 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:53.717360 14685 ts_tablet_manager.cc:1875] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.717602 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table_new_name [id=1dd60d66aa1648d783009ecca1586da8]) successfully deleted
I20240501 07:58:53.719105 14683 ts_tablet_manager.cc:1875] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:53.725708 14683 ts_tablet_manager.cc:1888] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.725791 14683 log.cc:1196] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f46a78e76be24ababa66b6f5eb92c696
I20240501 07:58:53.730285 14685 ts_tablet_manager.cc:1888] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:53.730664 14683 ts_tablet_manager.cc:1909] T f46a78e76be24ababa66b6f5eb92c696 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:53.730796 14685 log.cc:1196] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f46a78e76be24ababa66b6f5eb92c696
I20240501 07:58:53.731175 14685 ts_tablet_manager.cc:1909] T f46a78e76be24ababa66b6f5eb92c696 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:53.732210 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table_new_name [id=1dd60d66aa1648d783009ecca1586da8]) successfully deleted
I20240501 07:58:53.733240 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet f46a78e76be24ababa66b6f5eb92c696 (table rebalancer_dynamic_table_new_name [id=1dd60d66aa1648d783009ecca1586da8]) successfully deleted
I20240501 07:58:53.734990 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:53.737341 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:53.740320 12648 consensus_queue.cc:1035] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.741058 14505 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.742483 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.750610 14505 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.752137 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:53.756088 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.757630 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:53.759163 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.773955 14080 consensus_queue.cc:1035] T f4729eaaa94e480ea9bf075a355dffa7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.797609 9250 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:53.802942 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:53.809794 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 681fe317204c41218538e33c9a14bc72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]), partition=
I20240501 07:58:53.809890 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 681fe317204c41218538e33c9a14bc72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]), partition=
I20240501 07:58:53.810053 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681fe317204c41218538e33c9a14bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.810114 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681fe317204c41218538e33c9a14bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.812729 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 681fe317204c41218538e33c9a14bc72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]), partition=
I20240501 07:58:53.812958 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681fe317204c41218538e33c9a14bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:53.820926 14706 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:53.821940 14706 tablet_bootstrap.cc:654] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.822533 14705 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:53.822574 14707 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:53.823532 14705 tablet_bootstrap.cc:654] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.823616 14707 tablet_bootstrap.cc:654] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:53.825817 14706 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:53.825866 14705 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:53.825960 14705 ts_tablet_manager.cc:1367] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:53.825992 14706 ts_tablet_manager.cc:1367] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:53.826407 14705 raft_consensus.cc:348] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.826527 14705 raft_consensus.cc:374] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.826565 14705 raft_consensus.cc:729] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.826675 14705 consensus_queue.cc:260] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.826929 14705 ts_tablet_manager.cc:1398] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:53.826838 14706 raft_consensus.cc:348] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.828240 14706 raft_consensus.cc:374] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.828334 14706 raft_consensus.cc:729] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.828555 14706 consensus_queue.cc:260] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.830062 14707 tablet_bootstrap.cc:492] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:53.830163 14707 ts_tablet_manager.cc:1367] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:53.830628 14707 raft_consensus.cc:348] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.830752 14707 raft_consensus.cc:374] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:53.830786 14707 raft_consensus.cc:729] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:53.832412 14706 ts_tablet_manager.cc:1398] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:58:53.830912 14707 consensus_queue.cc:260] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.832868 14707 ts_tablet_manager.cc:1398] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:53.848435 6877 consensus_queue.cc:237] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.849732 7140 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:53.849789 9665 raft_consensus.cc:1261] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:53.850198 14141 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:53.850636 14700 consensus_queue.cc:1035] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:53.852669 14141 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.853075 7140 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.853562 9665 raft_consensus.cc:2932] T b6421aafe6b34136bd8e1853cdf50786 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.856565 6732 catalog_manager.cc:5484] T b6421aafe6b34136bd8e1853cdf50786 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.859359 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet b6421aafe6b34136bd8e1853cdf50786 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:53.859598 14646 tablet_replica.cc:330] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:53.859773 14646 raft_consensus.cc:2229] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.859920 14646 raft_consensus.cc:2258] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.860445 14646 ts_tablet_manager.cc:1875] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:53.862119 14646 ts_tablet_manager.cc:1888] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:53.862186 14646 log.cc:1196] T b6421aafe6b34136bd8e1853cdf50786 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b6421aafe6b34136bd8e1853cdf50786
I20240501 07:58:53.862646 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet b6421aafe6b34136bd8e1853cdf50786 (table rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]) successfully deleted
I20240501 07:58:53.946309 13301 raft_consensus.cc:482] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:53.946450 13301 raft_consensus.cc:504] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.947144 13301 leader_election.cc:288] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:53.947348 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681fe317204c41218538e33c9a14bc72" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:53.947497 6877 raft_consensus.cc:2454] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:53.947754 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681fe317204c41218538e33c9a14bc72" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:53.947799 7207 leader_election.cc:302] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:53.947871 9665 raft_consensus.cc:2454] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:53.947948 13301 raft_consensus.cc:2781] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:53.948006 13301 raft_consensus.cc:482] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:53.948030 13301 raft_consensus.cc:3037] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.948933 13301 raft_consensus.cc:504] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.949227 13301 leader_election.cc:288] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:53.949560 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681fe317204c41218538e33c9a14bc72" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:53.949600 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681fe317204c41218538e33c9a14bc72" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:53.949707 9665 raft_consensus.cc:3037] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.949786 6877 raft_consensus.cc:3037] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:53.950516 9665 raft_consensus.cc:2454] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:53.950691 6877 raft_consensus.cc:2454] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:53.951083 7207 leader_election.cc:302] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162; no voters:
I20240501 07:58:53.951282 13301 raft_consensus.cc:2781] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:53.951457 13301 raft_consensus.cc:686] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:53.951625 13301 consensus_queue.cc:237] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:53.953418 6735 catalog_manager.cc:5484] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:53.970952 7252 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:53.972623 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.975708 9250 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:53.980751 13577 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.981534 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:53.981600 9382 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:53.981600 9507 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:53.986418 14505 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:53.991650 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:54.000519 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 791b17b0049c4bbbbf56934dcfe2f742 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.000511 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 791b17b0049c4bbbbf56934dcfe2f742 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.000818 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791b17b0049c4bbbbf56934dcfe2f742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.000823 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791b17b0049c4bbbbf56934dcfe2f742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.001461 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 791b17b0049c4bbbbf56934dcfe2f742 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.001705 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791b17b0049c4bbbbf56934dcfe2f742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.002810 14705 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:54.003257 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2044f11ba34d6f9225781c0b8fdb51 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.003564 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2044f11ba34d6f9225781c0b8fdb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.003978 14705 tablet_bootstrap.cc:654] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.004787 6989 tablet_service.cc:1467] Processing CreateTablet for tablet b78be08b353141469d49c7460393a051 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.004817 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2044f11ba34d6f9225781c0b8fdb51 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.005035 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2044f11ba34d6f9225781c0b8fdb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.005061 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78be08b353141469d49c7460393a051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.005719 14707 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:54.005892 9381 tablet_service.cc:1467] Processing CreateTablet for tablet b78be08b353141469d49c7460393a051 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.006132 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78be08b353141469d49c7460393a051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.006767 14707 tablet_bootstrap.cc:654] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.007426 9250 tablet_service.cc:1467] Processing CreateTablet for tablet b78be08b353141469d49c7460393a051 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.007716 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78be08b353141469d49c7460393a051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.013063 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2044f11ba34d6f9225781c0b8fdb51 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.013337 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2044f11ba34d6f9225781c0b8fdb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.015087 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:54.020040 14789 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:54.021345 14789 tablet_bootstrap.cc:654] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.022013 14785 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:54.023423 14705 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:54.023514 14705 ts_tablet_manager.cc:1367] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:58:54.024044 14705 raft_consensus.cc:348] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.024158 14705 raft_consensus.cc:374] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.024191 14705 raft_consensus.cc:729] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.024309 14705 consensus_queue.cc:260] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.024619 14705 ts_tablet_manager.cc:1398] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:54.024843 14705 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:54.025760 14705 tablet_bootstrap.cc:654] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.029112 14789 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:54.031124 14789 ts_tablet_manager.cc:1367] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:54.031613 14789 raft_consensus.cc:348] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.031724 14789 raft_consensus.cc:374] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.031760 14789 raft_consensus.cc:729] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.032063 14787 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:54.033528 14785 tablet_bootstrap.cc:654] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.034165 14789 consensus_queue.cc:260] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.035732 14789 ts_tablet_manager.cc:1398] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:54.035473 14785 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:54.036266 14787 tablet_bootstrap.cc:654] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.036844 14785 ts_tablet_manager.cc:1367] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:54.037467 14785 raft_consensus.cc:348] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.037642 14785 raft_consensus.cc:374] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.037716 14785 raft_consensus.cc:729] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.038075 14785 consensus_queue.cc:260] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.039510 14785 ts_tablet_manager.cc:1398] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:54.039949 14785 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:54.040606 14705 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:54.040766 14705 ts_tablet_manager.cc:1367] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240501 07:58:54.041877 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.043066 14705 raft_consensus.cc:348] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.043176 14705 raft_consensus.cc:374] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.043359 14705 raft_consensus.cc:729] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.043541 14705 consensus_queue.cc:260] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.044013 14705 ts_tablet_manager.cc:1398] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:54.045668 14785 tablet_bootstrap.cc:654] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.046758 14787 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:54.046895 14787 ts_tablet_manager.cc:1367] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:54.047475 14787 raft_consensus.cc:348] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.047758 14787 raft_consensus.cc:374] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.047834 14787 raft_consensus.cc:729] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.048314 14787 consensus_queue.cc:260] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.048652 14787 ts_tablet_manager.cc:1398] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.049583 14785 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:54.049724 14785 ts_tablet_manager.cc:1367] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 07:58:54.050284 14785 raft_consensus.cc:348] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.050459 14785 raft_consensus.cc:374] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.050532 14785 raft_consensus.cc:729] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.050709 14785 consensus_queue.cc:260] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.051798 14785 ts_tablet_manager.cc:1398] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.063251 14784 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:54.064309 14784 tablet_bootstrap.cc:654] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.065014 14707 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:54.065107 14707 ts_tablet_manager.cc:1367] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20240501 07:58:54.065680 14707 raft_consensus.cc:348] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.065793 14707 raft_consensus.cc:374] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.065819 14707 raft_consensus.cc:729] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.065932 14707 consensus_queue.cc:260] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.066228 14707 ts_tablet_manager.cc:1398] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.070868 14784 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:54.071054 14784 ts_tablet_manager.cc:1367] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:54.071063 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:54.072230 14784 raft_consensus.cc:348] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.072463 14784 raft_consensus.cc:374] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.072535 14784 raft_consensus.cc:729] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.072739 14784 consensus_queue.cc:260] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.073230 14784 ts_tablet_manager.cc:1398] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.073444 14784 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:54.075692 14784 tablet_bootstrap.cc:654] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.077548 9665 raft_consensus.cc:1261] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.077634 6877 raft_consensus.cc:1261] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.077785 14784 tablet_bootstrap.cc:492] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:54.077868 14784 ts_tablet_manager.cc:1367] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:54.078490 14622 consensus_queue.cc:1035] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.078673 14784 raft_consensus.cc:348] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.078756 14784 raft_consensus.cc:374] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.078819 14784 raft_consensus.cc:729] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.078830 14622 consensus_queue.cc:1035] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.078924 14784 consensus_queue.cc:260] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.079203 14784 ts_tablet_manager.cc:1398] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.080794 12912 raft_consensus.cc:482] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.080886 12912 raft_consensus.cc:504] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.081225 12912 leader_election.cc:288] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:54.081612 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:54.081645 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:54.081758 7010 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:54.081758 9266 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:54.081981 9339 leader_election.cc:302] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:54.082249 12912 raft_consensus.cc:2781] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.082305 12912 raft_consensus.cc:482] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.082335 12912 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.083081 6732 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]). Expected version 1 got 0
I20240501 07:58:54.083325 12912 raft_consensus.cc:504] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.083709 12912 leader_election.cc:288] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:54.083762 6735 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]). Expected version 1 got 0
I20240501 07:58:54.085354 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:54.085498 7010 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.085835 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:54.085966 9266 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.086565 7010 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:54.086952 9339 leader_election.cc:302] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:54.087121 9266 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:54.087211 12912 raft_consensus.cc:2781] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.087294 12912 raft_consensus.cc:686] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:54.087464 12912 consensus_queue.cc:237] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.089802 14802 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.090092 14802 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:54.090332 6734 catalog_manager.cc:5484] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:54.093081 6735 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]). Expected version 1 got 0
I20240501 07:58:54.094063 14805 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.094985 6729 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234]). Expected version 1 got 0
I20240501 07:58:54.095144 14804 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.095580 14804 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:54.098771 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234] alter complete (version 1)
I20240501 07:58:54.099670 14805 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:54.117060 7010 consensus_queue.cc:237] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:54.118322 7272 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:54.118495 9665 raft_consensus.cc:1261] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:54.118727 14080 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:54.118899 12648 consensus_queue.cc:1035] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:54.120780 14824 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.121122 9665 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.121183 7272 raft_consensus.cc:2932] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.123878 6729 catalog_manager.cc:5484] T c2f299abad9e415283470b4b214fc62f P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.126945 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f299abad9e415283470b4b214fc62f with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:54.127544 14825 tablet_replica.cc:330] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:54.127686 14825 raft_consensus.cc:2229] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:54.127823 14825 raft_consensus.cc:2258] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.128360 14825 ts_tablet_manager.cc:1875] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:54.130278 14825 ts_tablet_manager.cc:1888] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:54.130409 14825 log.cc:1196] T c2f299abad9e415283470b4b214fc62f P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2f299abad9e415283470b4b214fc62f
I20240501 07:58:54.130987 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet c2f299abad9e415283470b4b214fc62f (table rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]) successfully deleted
I20240501 07:58:54.142011 14791 raft_consensus.cc:482] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.142117 14791 raft_consensus.cc:504] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.142527 14791 leader_election.cc:288] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.143308 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:54.143391 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:54.143450 7272 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:54.143527 9665 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:54.143790 9469 leader_election.cc:302] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:54.144037 14116 raft_consensus.cc:2781] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.144079 14116 raft_consensus.cc:482] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.144114 14116 raft_consensus.cc:3037] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.145159 14116 raft_consensus.cc:504] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.145493 14116 leader_election.cc:288] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.145865 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:54.145965 7270 raft_consensus.cc:3037] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.146862 7270 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:54.148150 9469 leader_election.cc:302] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:54.148206 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:54.148313 9665 raft_consensus.cc:3037] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.148613 14791 raft_consensus.cc:2781] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.148703 14791 raft_consensus.cc:686] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:54.148814 14791 consensus_queue.cc:237] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.149281 9665 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:54.150923 6732 catalog_manager.cc:5484] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.156927 14531 raft_consensus.cc:482] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.157017 14531 raft_consensus.cc:504] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.157346 14531 leader_election.cc:288] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.157676 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791b17b0049c4bbbbf56934dcfe2f742" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:54.157799 9534 raft_consensus.cc:2454] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:54.158004 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791b17b0049c4bbbbf56934dcfe2f742" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:54.158121 7272 raft_consensus.cc:2454] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:54.158114 9339 leader_election.cc:302] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:54.158504 14531 raft_consensus.cc:2781] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.158561 14531 raft_consensus.cc:482] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.158602 14531 raft_consensus.cc:3037] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.159626 14531 raft_consensus.cc:504] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.159966 14531 leader_election.cc:288] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.160657 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791b17b0049c4bbbbf56934dcfe2f742" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:54.160787 9534 raft_consensus.cc:3037] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.161868 9534 raft_consensus.cc:2454] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:54.162135 9339 leader_election.cc:302] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:54.162397 14531 raft_consensus.cc:2781] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.162475 14531 raft_consensus.cc:686] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:54.162587 14531 consensus_queue.cc:237] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:54.162611 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791b17b0049c4bbbbf56934dcfe2f742" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:54.162720 7272 raft_consensus.cc:3037] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.164544 7272 raft_consensus.cc:2454] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:54.164566 6732 catalog_manager.cc:5484] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:54.205256 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:54.216290 14802 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:54.216636 14802 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:54.217815 14804 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:54.218132 14804 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:54.223084 14805 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:54.223417 14805 tablet.cc:1620] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:54.231444 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=eea91193a0db4325a9a63fce5cc44234] alter complete (version 2)
I20240501 07:58:54.295318 6859 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:54.295370 9250 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:54.304908 9507 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:54.308195 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.308488 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:54.334329 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:54.340579 9381 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:54.341605 14802 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:54.343811 14804 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:54.349658 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=eea91193a0db4325a9a63fce5cc44234] alter complete (version 3)
I20240501 07:58:54.353499 14805 tablet.cc:1722] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:54.376807 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.400822 7252 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:54.452818 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:54.452992 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:54.456912 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 681fe317204c41218538e33c9a14bc72
I20240501 07:58:54.457538 6859 tablet_service.cc:1514] Processing DeleteTablet for tablet 681fe317204c41218538e33c9a14bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:54 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:54.457616 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 681fe317204c41218538e33c9a14bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:54 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:54.457871 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 681fe317204c41218538e33c9a14bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:54 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:54.458009 14919 tablet_replica.cc:330] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:54.458168 14919 raft_consensus.cc:2229] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:54.458216 14920 tablet_replica.cc:330] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:54.458494 14919 raft_consensus.cc:2258] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.458547 14920 raft_consensus.cc:2229] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:54.458739 14920 raft_consensus.cc:2258] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.459118 14918 tablet_replica.cc:330] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:54.459282 14918 raft_consensus.cc:2229] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:54.459522 14918 raft_consensus.cc:2258] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.459744 6876 consensus_queue.cc:237] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:54.459892 14920 ts_tablet_manager.cc:1875] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:54.460253 14918 ts_tablet_manager.cc:1875] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:54.461038 7140 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:54.461828 9534 raft_consensus.cc:1261] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:54.461860 14141 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:54.462311 14141 consensus_queue.cc:1035] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:54.462924 14919 ts_tablet_manager.cc:1875] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:54.463138 14918 ts_tablet_manager.cc:1888] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:54.463287 14918 log.cc:1196] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/681fe317204c41218538e33c9a14bc72
I20240501 07:58:54.463745 14918 ts_tablet_manager.cc:1909] T 681fe317204c41218538e33c9a14bc72 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:54.464293 14920 ts_tablet_manager.cc:1888] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:54.464409 14920 log.cc:1196] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/681fe317204c41218538e33c9a14bc72
I20240501 07:58:54.464612 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table_new_name [id=eea91193a0db4325a9a63fce5cc44234]) successfully deleted
I20240501 07:58:54.464856 14920 ts_tablet_manager.cc:1909] T 681fe317204c41218538e33c9a14bc72 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:54.465598 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table_new_name [id=eea91193a0db4325a9a63fce5cc44234]) successfully deleted
I20240501 07:58:54.465785 14919 ts_tablet_manager.cc:1888] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:54.465869 14919 log.cc:1196] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/681fe317204c41218538e33c9a14bc72
I20240501 07:58:54.466212 14919 ts_tablet_manager.cc:1909] T 681fe317204c41218538e33c9a14bc72 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:54.466635 14141 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:54.467691 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 681fe317204c41218538e33c9a14bc72 (table rebalancer_dynamic_table_new_name [id=eea91193a0db4325a9a63fce5cc44234]) successfully deleted
I20240501 07:58:54.467990 7140 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:54.468253 9534 raft_consensus.cc:2932] T 69ff85c89fa443539b28c0d0cc299cfd P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:54.470749 6733 catalog_manager.cc:5484] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 4 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:58:54.473323 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 69ff85c89fa443539b28c0d0cc299cfd with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:54.473963 12912 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.474038 14928 tablet_replica.cc:330] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:54.474193 14928 raft_consensus.cc:2229] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:54.474483 14928 raft_consensus.cc:2258] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.474994 14928 ts_tablet_manager.cc:1875] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:54.477531 12912 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.477887 14928 ts_tablet_manager.cc:1888] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:58:54.477949 14928 log.cc:1196] T 69ff85c89fa443539b28c0d0cc299cfd P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/69ff85c89fa443539b28c0d0cc299cfd
I20240501 07:58:54.479032 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 69ff85c89fa443539b28c0d0cc299cfd (table rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]) successfully deleted
I20240501 07:58:54.526111 9666 consensus_queue.cc:237] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.527428 9534 raft_consensus.cc:1261] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.527716 9266 raft_consensus.cc:1261] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.528146 14971 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:54.528357 13577 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:54.528689 9601 consensus_peers.cc:485] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64e235bbc64a4dca8028ebf89c2b7b5e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.531005 13577 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:54.531324 9534 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:54.533025 9266 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:54.535694 6728 catalog_manager.cc:5484] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:54.558805 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:54.563783 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 6a58872a302445299d0de85b4d1d852c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]), partition=
I20240501 07:58:54.563942 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 6a58872a302445299d0de85b4d1d852c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]), partition=
I20240501 07:58:54.563942 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 6a58872a302445299d0de85b4d1d852c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]), partition=
I20240501 07:58:54.564028 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a58872a302445299d0de85b4d1d852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.564173 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a58872a302445299d0de85b4d1d852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.564173 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a58872a302445299d0de85b4d1d852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.566334 14989 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:54.566488 14707 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:54.567374 14990 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:54.567468 14707 tablet_bootstrap.cc:654] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.568513 14989 tablet_bootstrap.cc:654] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.568702 14990 tablet_bootstrap.cc:654] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.570104 14990 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:54.570113 14707 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:54.570180 14707 ts_tablet_manager.cc:1367] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:54.570184 14990 ts_tablet_manager.cc:1367] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:54.570636 14707 raft_consensus.cc:348] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.570636 14990 raft_consensus.cc:348] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.570735 14707 raft_consensus.cc:374] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.570735 14990 raft_consensus.cc:374] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.570767 14707 raft_consensus.cc:729] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.570767 14990 raft_consensus.cc:729] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.570883 14990 consensus_queue.cc:260] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.570883 14707 consensus_queue.cc:260] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.571183 14990 ts_tablet_manager.cc:1398] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.571182 14707 ts_tablet_manager.cc:1398] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.573704 14531 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.580142 12912 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.580278 7010 consensus_queue.cc:237] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
W20240501 07:58:54.582150 6945 consensus_peers.cc:485] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93af29aad2ff498dbd8bccaba12f6a70. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.582263 7272 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:54.582341 7140 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:54.582788 14824 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:54.583628 14824 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:54.586491 14824 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:54.585537 14989 tablet_bootstrap.cc:492] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:54.586824 14989 ts_tablet_manager.cc:1367] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:58:54.587337 14989 raft_consensus.cc:348] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.587462 14989 raft_consensus.cc:374] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.587496 14989 raft_consensus.cc:729] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.587656 14989 consensus_queue.cc:260] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.588598 14989 ts_tablet_manager.cc:1398] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.588539 7140 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:54.590553 7272 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:54.590694 6737 catalog_manager.cc:5484] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 4, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:54.612039 7140 consensus_queue.cc:237] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
W20240501 07:58:54.613348 7076 consensus_peers.cc:485] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2fa7739592145d5b90d98fecdbe02b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.613406 6876 raft_consensus.cc:1261] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:54.613762 7010 raft_consensus.cc:1261] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:54.613807 15018 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:54.614640 15018 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:54.616338 14450 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.616739 7010 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.618461 6876 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.619114 6737 catalog_manager.cc:5484] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:54.639931 9402 consensus_queue.cc:237] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.641343 9271 raft_consensus.cc:1261] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.641391 9666 raft_consensus.cc:1261] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:54.641655 9337 consensus_peers.cc:485] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c72ca77c85c94540905f236a4fc2e300. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.641746 12912 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:54.641920 15038 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:54.643772 12912 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.644153 9666 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.648011 6733 catalog_manager.cc:5484] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.648609 9271 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.667212 9271 consensus_queue.cc:237] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:54.668493 9534 raft_consensus.cc:1261] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.668699 9402 raft_consensus.cc:1261] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.668885 14901 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:54.669044 14900 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:54.669178 9205 consensus_peers.cc:485] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae59405a605c4b30a508002a9df30945. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.670946 14900 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:54.671427 9534 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:54.674474 6733 catalog_manager.cc:5484] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:54.674567 9402 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:54.678001 14791 consensus_queue.cc:1035] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.678633 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:54.683550 14116 consensus_queue.cc:1035] T 4e2044f11ba34d6f9225781c0b8fdb51 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.685432 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 03a798f72da143c3a448e918eb594445 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.685364 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1605ddfb2c4606aacaf28184c37396 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.685710 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a798f72da143c3a448e918eb594445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.685932 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1605ddfb2c4606aacaf28184c37396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.686784 9381 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.687074 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.687096 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1605ddfb2c4606aacaf28184c37396 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.687366 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1605ddfb2c4606aacaf28184c37396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.688344 6859 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.688545 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.689702 9250 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.689970 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb82e1b5bb40c4bdd1d27cb1cb5e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.690541 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 03a798f72da143c3a448e918eb594445 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.690765 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a798f72da143c3a448e918eb594445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.691874 14989 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:54.692018 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 03a798f72da143c3a448e918eb594445 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.692193 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a798f72da143c3a448e918eb594445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.692764 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1605ddfb2c4606aacaf28184c37396 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=0f3dc8e0a03a4aa6b0b2cf05b15747b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.692940 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1605ddfb2c4606aacaf28184c37396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.693176 14989 tablet_bootstrap.cc:654] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.694991 15067 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:54.695974 15067 tablet_bootstrap.cc:654] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.696583 15068 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:54.697118 15069 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:54.698179 15068 tablet_bootstrap.cc:654] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.698606 15067 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:54.699106 15067 ts_tablet_manager.cc:1367] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:54.699540 15067 raft_consensus.cc:348] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.699991 15067 raft_consensus.cc:374] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.700079 15067 raft_consensus.cc:729] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.700258 15067 consensus_queue.cc:260] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.700879 15068 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:54.700990 15068 ts_tablet_manager.cc:1367] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:54.701233 15067 ts_tablet_manager.cc:1398] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:54.701906 15068 raft_consensus.cc:348] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.702023 15068 raft_consensus.cc:374] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.702067 15068 raft_consensus.cc:729] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.702190 15068 consensus_queue.cc:260] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.702558 15068 ts_tablet_manager.cc:1398] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:54.702790 15067 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:54.703641 15067 tablet_bootstrap.cc:654] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.704743 15067 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:54.704831 15067 ts_tablet_manager.cc:1367] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.705143 15067 raft_consensus.cc:348] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.705222 15067 raft_consensus.cc:374] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.705258 15067 raft_consensus.cc:729] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.705410 15067 consensus_queue.cc:260] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.705698 15067 ts_tablet_manager.cc:1398] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.719763 15069 tablet_bootstrap.cc:654] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.721076 14989 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:54.721186 14989 ts_tablet_manager.cc:1367] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 07:58:54.721701 14989 raft_consensus.cc:348] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.721812 14989 raft_consensus.cc:374] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.721850 14989 raft_consensus.cc:729] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.721983 14989 consensus_queue.cc:260] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.722244 14989 ts_tablet_manager.cc:1398] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.722431 14989 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:54.723428 14989 tablet_bootstrap.cc:654] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.724993 15069 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:54.725137 15069 ts_tablet_manager.cc:1367] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 07:58:54.725733 15069 raft_consensus.cc:348] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.725916 15069 raft_consensus.cc:374] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.726003 15069 raft_consensus.cc:729] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.726155 15069 consensus_queue.cc:260] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.727643 15038 raft_consensus.cc:482] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.727741 15038 raft_consensus.cc:504] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.728147 15038 leader_election.cc:288] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:54.728596 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb82e1b5bb40c4bdd1d27cb1cb5e99" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:54.728771 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb82e1b5bb40c4bdd1d27cb1cb5e99" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:54.728909 9271 raft_consensus.cc:2454] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
W20240501 07:58:54.729259 9338 leader_election.cc:341] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:54.729377 9338 leader_election.cc:302] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters: 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:54.729516 15038 raft_consensus.cc:2781] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.729583 15038 raft_consensus.cc:482] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.729626 15038 raft_consensus.cc:3037] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.730746 15038 raft_consensus.cc:504] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.731124 15038 leader_election.cc:288] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:54.731456 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb82e1b5bb40c4bdd1d27cb1cb5e99" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:54.731561 9271 raft_consensus.cc:3037] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.732564 9271 raft_consensus.cc:2454] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:54.732484 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb82e1b5bb40c4bdd1d27cb1cb5e99" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
W20240501 07:58:54.732971 9338 leader_election.cc:341] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:54.733160 9338 leader_election.cc:302] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters: 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:54.729401 15069 ts_tablet_manager.cc:1398] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.733340 15038 raft_consensus.cc:2781] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.733420 15038 raft_consensus.cc:686] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:54.733531 15038 consensus_queue.cc:237] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.735445 15069 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:54.736075 6735 catalog_manager.cc:5484] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:54.736305 14989 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:54.736394 14989 ts_tablet_manager.cc:1367] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20240501 07:58:54.736966 14989 raft_consensus.cc:348] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.737075 14989 raft_consensus.cc:374] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.737112 14989 raft_consensus.cc:729] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.737274 14989 consensus_queue.cc:260] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.737561 14989 ts_tablet_manager.cc:1398] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:58:54.737726 14989 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:54.738655 14989 tablet_bootstrap.cc:654] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.739457 15069 tablet_bootstrap.cc:654] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.744915 14989 tablet_bootstrap.cc:492] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:54.744997 14989 ts_tablet_manager.cc:1367] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 07:58:54.745492 14989 raft_consensus.cc:348] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.745581 14989 raft_consensus.cc:374] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.745613 14989 raft_consensus.cc:729] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.745713 14989 consensus_queue.cc:260] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.745967 14989 ts_tablet_manager.cc:1398] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:54.751739 15069 tablet_bootstrap.cc:492] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:54.751818 15069 ts_tablet_manager.cc:1367] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:58:54.752260 15069 raft_consensus.cc:348] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.752367 15069 raft_consensus.cc:374] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.752398 15069 raft_consensus.cc:729] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.752506 15069 consensus_queue.cc:260] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:54.752784 15069 ts_tablet_manager.cc:1398] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.752946 15069 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:54.753948 15069 tablet_bootstrap.cc:654] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.756063 15069 tablet_bootstrap.cc:492] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:54.756191 15069 ts_tablet_manager.cc:1367] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:54.756611 15069 raft_consensus.cc:348] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.756691 15069 raft_consensus.cc:374] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.756726 15069 raft_consensus.cc:729] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.756870 15069 consensus_queue.cc:260] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.757177 15069 ts_tablet_manager.cc:1398] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.795899 13806 raft_consensus.cc:482] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.796005 13806 raft_consensus.cc:504] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.796408 13806 leader_election.cc:288] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:54.796788 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a58872a302445299d0de85b4d1d852c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:54.796921 7010 raft_consensus.cc:2454] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.797125 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a58872a302445299d0de85b4d1d852c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:54.797243 9666 raft_consensus.cc:2454] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.797571 9207 leader_election.cc:302] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:54.797744 14901 raft_consensus.cc:2781] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.797808 14901 raft_consensus.cc:482] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.797855 14901 raft_consensus.cc:3037] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.798983 14901 raft_consensus.cc:504] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.799357 14901 leader_election.cc:288] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:54.799754 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a58872a302445299d0de85b4d1d852c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:54.799824 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a58872a302445299d0de85b4d1d852c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:54.799865 7010 raft_consensus.cc:3037] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.799932 9666 raft_consensus.cc:3037] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.800783 7010 raft_consensus.cc:2454] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.800967 9666 raft_consensus.cc:2454] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.801129 9207 leader_election.cc:302] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:54.801663 14901 raft_consensus.cc:2781] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.801750 14901 raft_consensus.cc:686] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:54.801864 14901 consensus_queue.cc:237] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:54.805037 6732 catalog_manager.cc:5484] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:54.807137 14901 raft_consensus.cc:482] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.807222 14901 raft_consensus.cc:504] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.807554 14901 leader_election.cc:288] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.808100 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1605ddfb2c4606aacaf28184c37396" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:54.808245 7272 raft_consensus.cc:2454] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.808473 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1605ddfb2c4606aacaf28184c37396" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:54.808576 9205 leader_election.cc:302] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:54.808599 6876 raft_consensus.cc:2454] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.808727 14901 raft_consensus.cc:2781] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.808768 14901 raft_consensus.cc:482] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.808807 14901 raft_consensus.cc:3037] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.811410 14901 raft_consensus.cc:504] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.812003 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1605ddfb2c4606aacaf28184c37396" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:54.812134 6876 raft_consensus.cc:3037] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.813145 6876 raft_consensus.cc:2454] T 6a1605ddfb2c4606aacaf28184c37396 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.813519 9206 leader_election.cc:302] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:54.813683 13806 raft_consensus.cc:2781] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.813782 13806 raft_consensus.cc:686] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:54.813912 13806 consensus_queue.cc:237] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.816030 6732 catalog_manager.cc:5484] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.816120 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1605ddfb2c4606aacaf28184c37396" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:54.816231 7272 raft_consensus.cc:3037] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.817160 7272 raft_consensus.cc:2454] T 6a1605ddfb2c4606aacaf28184c37396 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.818733 14901 leader_election.cc:288] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.874890 14901 raft_consensus.cc:482] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.875059 14901 raft_consensus.cc:504] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.875384 14901 leader_election.cc:288] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.875741 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a798f72da143c3a448e918eb594445" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:54.875751 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a798f72da143c3a448e918eb594445" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:54.875870 7272 raft_consensus.cc:2454] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.875873 6876 raft_consensus.cc:2454] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:54.876240 9205 leader_election.cc:302] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:54.876421 14901 raft_consensus.cc:2781] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.876492 14901 raft_consensus.cc:482] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.876529 14901 raft_consensus.cc:3037] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.877708 14901 raft_consensus.cc:504] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.878037 14901 leader_election.cc:288] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:54.878324 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a798f72da143c3a448e918eb594445" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:54.878366 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a798f72da143c3a448e918eb594445" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:54.878425 7272 raft_consensus.cc:3037] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.878508 6876 raft_consensus.cc:3037] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.879388 6876 raft_consensus.cc:2454] T 03a798f72da143c3a448e918eb594445 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.879400 7272 raft_consensus.cc:2454] T 03a798f72da143c3a448e918eb594445 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:54.879671 9205 leader_election.cc:302] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:54.879848 14901 raft_consensus.cc:2781] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.880016 14901 raft_consensus.cc:686] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:54.880167 14901 consensus_queue.cc:237] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:54.882328 6728 catalog_manager.cc:5484] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:54.910331 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:54.912374 9238 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.912963 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:54.920171 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:54.920818 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.920985 9666 raft_consensus.cc:1261] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.921409 7010 raft_consensus.cc:1261] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:54.921386 13806 consensus_queue.cc:1035] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.921998 13806 consensus_queue.cc:1035] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:54.925133 6729 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]). Expected version 1 got 0
I20240501 07:58:54.925755 15126 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.926528 14934 mvcc.cc:205] Tried to move back new op lower bound from 7022798171832147968 to 7022798171349774336. Current Snapshot: MvccSnapshot[applied={T|T < 7022798171832147968}]
I20240501 07:58:54.927417 15126 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:54.929118 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:54.930383 7120 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:54.933365 6730 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]). Expected version 1 got 0
I20240501 07:58:54.933934 9381 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:54.939803 6730 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]). Expected version 1 got 0
I20240501 07:58:54.943563 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc] alter complete (version 1)
I20240501 07:58:54.948228 9507 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:54.979175 15127 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.981489 15127 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:54.984544 6728 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc]). Expected version 1 got 0
I20240501 07:58:54.989598 15129 ts_tablet_manager.cc:902] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:54.990083 15129 tablet_copy_client.cc:308] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:58:54.990594 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 64e235bbc64a4dca8028ebf89c2b7b5e from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:41375)
I20240501 07:58:54.990695 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 64e235bbc64a4dca8028ebf89c2b7b5e from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:41375: session id = 1a9029bee6a144938f68b363e4cca162-64e235bbc64a4dca8028ebf89c2b7b5e
I20240501 07:58:54.991676 9686 tablet_copy_source_session.cc:205] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:54.992494 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e235bbc64a4dca8028ebf89c2b7b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.992544 15128 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:54.992826 15128 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:54.996863 15129 tablet_copy_client.cc:783] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:54.997025 15129 tablet_copy_client.cc:647] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:54.998317 15129 tablet_copy_client.cc:523] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:54.999630 15129 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:55.003229 15129 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.003465 15129 tablet_bootstrap.cc:492] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:58:55.003607 15129 ts_tablet_manager.cc:1367] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 07:58:55.004050 15129 raft_consensus.cc:348] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:55.004165 15129 raft_consensus.cc:729] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.004318 15129 consensus_queue.cc:260] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:55.004647 15129 ts_tablet_manager.cc:1398] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.005156 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-64e235bbc64a4dca8028ebf89c2b7b5e received from {username='slave'} at 127.0.99.129:41375
I20240501 07:58:55.005231 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-64e235bbc64a4dca8028ebf89c2b7b5e on tablet 64e235bbc64a4dca8028ebf89c2b7b5e with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:55.037493 15140 ts_tablet_manager.cc:902] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:55.038075 15140 tablet_copy_client.cc:308] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:55.038537 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet c2fa7739592145d5b90d98fecdbe02b2 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:38089)
I20240501 07:58:55.038650 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet c2fa7739592145d5b90d98fecdbe02b2 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:38089: session id = 7fa3a88c25964445975bb47e0feed460-c2fa7739592145d5b90d98fecdbe02b2
I20240501 07:58:55.039695 7160 tablet_copy_source_session.cc:205] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.041014 15140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fa7739592145d5b90d98fecdbe02b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.043363 15140 tablet_copy_client.cc:783] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.043516 15140 tablet_copy_client.cc:647] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.045147 15140 tablet_copy_client.cc:523] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.046496 15140 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:55.059922 7140 consensus_queue.cc:237] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.061494 9534 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:55.061604 6876 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 07:58:55.061674 7077 consensus_peers.cc:485] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f334b66a84b94c3ba15004253b4cb714. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.061885 15017 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:55.062099 15018 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:55.062788 15157 ts_tablet_manager.cc:902] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:55.063364 15157 tablet_copy_client.cc:308] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:58:55.064069 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet c72ca77c85c94540905f236a4fc2e300 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:42663)
I20240501 07:58:55.064208 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet c72ca77c85c94540905f236a4fc2e300 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:42663: session id = 15ab96f6477b4291932409b80af07213-c72ca77c85c94540905f236a4fc2e300
I20240501 07:58:55.065479 9414 tablet_copy_source_session.cc:205] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.067839 15017 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.068125 15157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72ca77c85c94540905f236a4fc2e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.070114 15140 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.070204 6876 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.070390 15140 tablet_bootstrap.cc:492] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:55.070546 15140 ts_tablet_manager.cc:1367] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20240501 07:58:55.070828 15157 tablet_copy_client.cc:783] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.070964 15157 tablet_copy_client.cc:647] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.071053 15140 raft_consensus.cc:348] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:55.071148 15140 raft_consensus.cc:729] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.071259 15140 consensus_queue.cc:260] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:55.071585 15140 ts_tablet_manager.cc:1398] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:55.072764 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:55.073639 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-c2fa7739592145d5b90d98fecdbe02b2 received from {username='slave'} at 127.0.99.136:38089
I20240501 07:58:55.073696 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-c2fa7739592145d5b90d98fecdbe02b2 on tablet c2fa7739592145d5b90d98fecdbe02b2 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:55.073581 9534 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.074484 15157 tablet_copy_client.cc:523] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.077211 6730 catalog_manager.cc:5484] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 5 to 6, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.080771 15157 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:55.081449 15126 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.081804 15126 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:55.083091 15127 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.083359 15127 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:55.085542 15128 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.085820 15128 tablet.cc:1620] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:55.086695 15157 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.086923 15157 tablet_bootstrap.cc:492] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:58:55.087113 15157 ts_tablet_manager.cc:1367] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 07:58:55.087615 15157 raft_consensus.cc:348] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.087740 15157 raft_consensus.cc:729] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.087881 15157 consensus_queue.cc:260] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.088200 15157 ts_tablet_manager.cc:1398] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:55.090070 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-c72ca77c85c94540905f236a4fc2e300 received from {username='slave'} at 127.0.99.132:42663
I20240501 07:58:55.090190 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 15ab96f6477b4291932409b80af07213-c72ca77c85c94540905f236a4fc2e300 on tablet c72ca77c85c94540905f236a4fc2e300 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:58:55.091104 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c86a4593a4714b218e2b6e65faed9afc] alter complete (version 2)
I20240501 07:58:55.107430 9402 consensus_queue.cc:237] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:55.108681 7272 raft_consensus.cc:1261] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.109083 9534 raft_consensus.cc:1261] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:55.109282 9338 consensus_peers.cc:485] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3333244e06a4ad8bd3ecc73cb356c90. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.109315 15181 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.109756 12912 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.111624 14531 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:55.112069 9534 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:55.113744 7272 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:55.115737 6732 catalog_manager.cc:5484] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:55.128111 15194 ts_tablet_manager.cc:902] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:55.128742 15194 tablet_copy_client.cc:308] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:58:55.129225 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet ae59405a605c4b30a508002a9df30945 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:44503)
I20240501 07:58:55.129323 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet ae59405a605c4b30a508002a9df30945 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:44503: session id = fc946354b5d04503a56122e59fc320be-ae59405a605c4b30a508002a9df30945
I20240501 07:58:55.130311 9291 tablet_copy_source_session.cc:205] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.131182 15194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59405a605c4b30a508002a9df30945. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.133519 15194 tablet_copy_client.cc:783] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.133690 15194 tablet_copy_client.cc:647] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.134793 15194 tablet_copy_client.cc:523] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.135808 15194 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:55.138370 9665 consensus_queue.cc:237] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.139483 15194 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.139739 15194 tablet_bootstrap.cc:492] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:58:55.139899 15194 ts_tablet_manager.cc:1367] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 07:58:55.140045 7009 raft_consensus.cc:1261] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:55.140147 9601 consensus_peers.cc:485] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c684b44a722143c4a3c44caad7cd84be. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.140198 9402 raft_consensus.cc:1261] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.140408 15194 raft_consensus.cc:348] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.140513 15194 raft_consensus.cc:729] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.140728 14505 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.140714 15194 consensus_queue.cc:260] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.140890 15206 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.140993 15194 ts_tablet_manager.cc:1398] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:58:55.141399 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-ae59405a605c4b30a508002a9df30945 received from {username='slave'} at 127.0.99.131:44503
I20240501 07:58:55.141471 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session fc946354b5d04503a56122e59fc320be-ae59405a605c4b30a508002a9df30945 on tablet ae59405a605c4b30a508002a9df30945 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:58:55.148435 14505 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:55.149000 9402 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:55.150301 7009 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:55.151926 6728 catalog_manager.cc:5484] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:55.177609 15226 ts_tablet_manager.cc:902] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:55.178782 15226 tablet_copy_client.cc:308] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:58:55.179630 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 93af29aad2ff498dbd8bccaba12f6a70 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:37855)
I20240501 07:58:55.179740 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 93af29aad2ff498dbd8bccaba12f6a70 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:37855: session id = 49adb9e99822494196ed22044bfed81f-93af29aad2ff498dbd8bccaba12f6a70
I20240501 07:58:55.179935 9266 consensus_queue.cc:237] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.180680 7030 tablet_copy_source_session.cc:205] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.181416 9403 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:55.181550 9534 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:55.182109 14900 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:55.182993 15226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93af29aad2ff498dbd8bccaba12f6a70. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 07:58:55.183367 9205 consensus_peers.cc:485] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c5e1de6fc394fcb98c64165d74ca317. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.183584 13806 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:55.185096 15226 tablet_copy_client.cc:783] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.185245 15226 tablet_copy_client.cc:647] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.186811 14901 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.187278 9534 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.188582 9402 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.190460 6732 catalog_manager.cc:5484] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 3 to 4, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.192101 15226 tablet_copy_client.cc:523] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.193291 15226 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:55.195072 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:55.202018 14531 consensus_queue.cc:1035] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.202733 15126 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:55.205264 15128 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:55.207325 15127 tablet.cc:1722] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:55.212399 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=c86a4593a4714b218e2b6e65faed9afc] alter complete (version 3)
I20240501 07:58:55.222036 6876 raft_consensus.cc:3037] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.223992 15226 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.224071 14531 consensus_queue.cc:1035] T c5cb82e1b5bb40c4bdd1d27cb1cb5e99 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.224457 15226 tablet_bootstrap.cc:492] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:55.224846 15226 ts_tablet_manager.cc:1367] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.001s
I20240501 07:58:55.225597 15226 raft_consensus.cc:348] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.225840 15226 raft_consensus.cc:729] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.226037 15226 consensus_queue.cc:260] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.226495 15226 ts_tablet_manager.cc:1398] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.226941 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-93af29aad2ff498dbd8bccaba12f6a70 received from {username='slave'} at 127.0.99.133:37855
I20240501 07:58:55.227066 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 49adb9e99822494196ed22044bfed81f-93af29aad2ff498dbd8bccaba12f6a70 on tablet 93af29aad2ff498dbd8bccaba12f6a70 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:55.236279 9402 consensus_queue.cc:237] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
W20240501 07:58:55.237639 9338 consensus_peers.cc:485] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79ab7f97eb2e439f8c974636b5791824. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.237704 9665 raft_consensus.cc:1261] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.237710 9534 raft_consensus.cc:1261] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.238081 15038 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.238232 15181 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.239965 14531 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:55.240324 9666 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:55.241678 9534 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:55.243438 6735 catalog_manager.cc:5484] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:55.247520 14901 consensus_queue.cc:1035] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.252117 13806 consensus_queue.cc:1035] T 6a1605ddfb2c4606aacaf28184c37396 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.266327 9271 consensus_queue.cc:237] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:55.267714 7010 raft_consensus.cc:1261] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.267813 9534 raft_consensus.cc:1261] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:55.267823 9206 consensus_peers.cc:485] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a550051551640dcab9733948bb0966f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.268172 14900 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.268332 13806 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.270390 14900 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:55.270767 7010 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:55.272138 9534 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:55.273470 6732 catalog_manager.cc:5484] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:55.283325 13806 consensus_queue.cc:1035] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.286929 13806 consensus_queue.cc:1035] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.295974 9665 consensus_queue.cc:237] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
W20240501 07:58:55.297372 9600 consensus_peers.cc:485] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 748b249371a4464f88aeb0c4c8558998. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:55.297516 9271 raft_consensus.cc:1261] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.297533 9402 raft_consensus.cc:1261] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.297899 13577 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.298064 15206 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:55.300143 13577 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.300565 9271 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.302763 9402 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.303809 6729 catalog_manager.cc:5484] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:55.315333 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:55.315479 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:55.317855 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 6a58872a302445299d0de85b4d1d852c
I20240501 07:58:55.318338 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a58872a302445299d0de85b4d1d852c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:55 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:55.319126 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a58872a302445299d0de85b4d1d852c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:55 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:55.319440 15311 tablet_replica.cc:330] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:55.319561 15311 raft_consensus.cc:2229] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:55.319656 15311 raft_consensus.cc:2258] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:55.320205 15311 ts_tablet_manager.cc:1875] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:55.322285 15311 ts_tablet_manager.cc:1888] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:55.322355 15311 log.cc:1196] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6a58872a302445299d0de85b4d1d852c
I20240501 07:58:55.322625 15311 ts_tablet_manager.cc:1909] T 6a58872a302445299d0de85b4d1d852c P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:55.323204 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table_new_name [id=c86a4593a4714b218e2b6e65faed9afc]) successfully deleted
I20240501 07:58:55.323300 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a58872a302445299d0de85b4d1d852c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:55 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:55.323635 15312 tablet_replica.cc:330] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:55.323781 15312 raft_consensus.cc:2229] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:55.323910 15312 raft_consensus.cc:2258] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:55.324487 15312 ts_tablet_manager.cc:1875] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:55.326543 15312 ts_tablet_manager.cc:1888] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:55.326615 15312 log.cc:1196] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a58872a302445299d0de85b4d1d852c
I20240501 07:58:55.326913 15312 ts_tablet_manager.cc:1909] T 6a58872a302445299d0de85b4d1d852c P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:55.327181 15310 tablet_replica.cc:330] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:55.327364 15310 raft_consensus.cc:2229] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:55.327591 15310 raft_consensus.cc:2258] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:55.327672 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table_new_name [id=c86a4593a4714b218e2b6e65faed9afc]) successfully deleted
I20240501 07:58:55.328075 15310 ts_tablet_manager.cc:1875] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:55.329768 15310 ts_tablet_manager.cc:1888] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:55.329869 15310 log.cc:1196] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/6a58872a302445299d0de85b4d1d852c
I20240501 07:58:55.330128 15310 ts_tablet_manager.cc:1909] T 6a58872a302445299d0de85b4d1d852c P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:55.330797 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 6a58872a302445299d0de85b4d1d852c (table rebalancer_dynamic_table_new_name [id=c86a4593a4714b218e2b6e65faed9afc]) successfully deleted
I20240501 07:58:55.403635 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:55.410313 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd704e855744f2b1e95d125f282e5f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.410494 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd704e855744f2b1e95d125f282e5f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.410616 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd704e855744f2b1e95d125f282e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.410791 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd704e855744f2b1e95d125f282e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.411429 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 12f4efcd9e274be483079915323045ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.411721 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f4efcd9e274be483079915323045ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.412994 15356 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:55.413096 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd704e855744f2b1e95d125f282e5f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.413319 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd704e855744f2b1e95d125f282e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.414216 15356 tablet_bootstrap.cc:654] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.415452 15356 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:55.415714 15356 ts_tablet_manager.cc:1367] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:55.416136 9381 tablet_service.cc:1467] Processing CreateTablet for tablet dadf963e650a45bd978cb5f0f027ad5a (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.416254 9645 tablet_service.cc:1467] Processing CreateTablet for tablet dadf963e650a45bd978cb5f0f027ad5a (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.416386 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadf963e650a45bd978cb5f0f027ad5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.416343 15356 raft_consensus.cc:348] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.416548 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadf963e650a45bd978cb5f0f027ad5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.416586 15356 raft_consensus.cc:374] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.416621 15356 raft_consensus.cc:729] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.416725 15356 consensus_queue.cc:260] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.417064 15356 ts_tablet_manager.cc:1398] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.418578 15360 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:55.419257 15357 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:55.419906 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 12f4efcd9e274be483079915323045ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.420097 15358 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:55.420150 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f4efcd9e274be483079915323045ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.421195 7120 tablet_service.cc:1467] Processing CreateTablet for tablet dadf963e650a45bd978cb5f0f027ad5a (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.421265 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 12f4efcd9e274be483079915323045ad (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=eba89bb11c94422c9646adf7c6842712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.421427 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadf963e650a45bd978cb5f0f027ad5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.421634 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:55.424142 15357 tablet_bootstrap.cc:654] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.424364 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f4efcd9e274be483079915323045ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.425050 15358 tablet_bootstrap.cc:654] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.426352 15357 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:55.427826 15360 tablet_bootstrap.cc:654] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.428408 15357 ts_tablet_manager.cc:1367] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:55.429061 9238 tablet_service.cc:1467] Processing CreateTablet for tablet b59513da29ad42939fa2ee264abce9fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]), partition=
I20240501 07:58:55.429322 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59513da29ad42939fa2ee264abce9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.430188 15361 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:55.429399 7121 tablet_service.cc:1467] Processing CreateTablet for tablet b59513da29ad42939fa2ee264abce9fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]), partition=
I20240501 07:58:55.430732 9507 tablet_service.cc:1467] Processing CreateTablet for tablet b59513da29ad42939fa2ee264abce9fb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]), partition=
I20240501 07:58:55.430820 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59513da29ad42939fa2ee264abce9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.430928 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59513da29ad42939fa2ee264abce9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.432832 15358 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:55.433233 15356 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:55.433784 15357 raft_consensus.cc:348] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.433917 15357 raft_consensus.cc:374] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.433956 15357 raft_consensus.cc:729] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.434134 15358 ts_tablet_manager.cc:1367] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:58:55.434165 15356 tablet_bootstrap.cc:654] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.434635 15358 raft_consensus.cc:348] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.434890 15357 consensus_queue.cc:260] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.435220 15357 ts_tablet_manager.cc:1398] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 07:58:55.434751 15358 raft_consensus.cc:374] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.435303 15358 raft_consensus.cc:729] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.435521 15361 tablet_bootstrap.cc:654] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.436221 15357 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:55.436409 15358 consensus_queue.cc:260] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.437417 15357 tablet_bootstrap.cc:654] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.437777 15358 ts_tablet_manager.cc:1398] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:55.437992 15358 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:55.439085 15358 tablet_bootstrap.cc:654] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.439911 15360 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:55.440969 15360 ts_tablet_manager.cc:1367] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20240501 07:58:55.441525 15360 raft_consensus.cc:348] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.442798 15360 raft_consensus.cc:374] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.442895 15360 raft_consensus.cc:729] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.443131 15360 consensus_queue.cc:260] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.445160 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.445644 15366 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:55.446581 15366 tablet_bootstrap.cc:654] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.447839 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.450671 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.452653 15361 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:55.452811 15361 ts_tablet_manager.cc:1367] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 07:58:55.453306 15361 raft_consensus.cc:348] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.453718 15361 raft_consensus.cc:374] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.453794 15361 raft_consensus.cc:729] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.453976 15361 consensus_queue.cc:260] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.462263 9238 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.463878 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.464435 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.464783 15360 ts_tablet_manager.cc:1398] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240501 07:58:55.464949 15360 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:55.465704 15181 raft_consensus.cc:482] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.465801 15181 raft_consensus.cc:504] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.466125 15360 tablet_bootstrap.cc:654] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.466183 15181 leader_election.cc:288] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:55.466567 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:55.466908 15361 ts_tablet_manager.cc:1398] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 07:58:55.467197 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:55.467315 9666 raft_consensus.cc:2454] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
W20240501 07:58:55.468031 9337 leader_election.cc:341] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:55.471393 9338 leader_election.cc:302] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters: fc946354b5d04503a56122e59fc320be
I20240501 07:58:55.471596 15366 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:55.471704 15366 ts_tablet_manager.cc:1367] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:58:55.472771 15366 raft_consensus.cc:348] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.472885 15366 raft_consensus.cc:374] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.472921 15366 raft_consensus.cc:729] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.473039 15366 consensus_queue.cc:260] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.473299 15366 ts_tablet_manager.cc:1398] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.475462 15038 raft_consensus.cc:2781] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.475557 15038 raft_consensus.cc:482] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.475608 15038 raft_consensus.cc:3037] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.476724 15038 raft_consensus.cc:504] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.477104 15038 leader_election.cc:288] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:55.477962 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:55.478076 9666 raft_consensus.cc:3037] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.479149 9666 raft_consensus.cc:2454] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:55.479712 9338 leader_election.cc:302] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:55.481000 15181 raft_consensus.cc:2781] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.481086 15181 raft_consensus.cc:686] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:55.481031 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:55.481199 15181 consensus_queue.cc:237] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:58:55.482081 9337 leader_election.cc:341] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:55.482620 9507 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:55.486131 6729 catalog_manager.cc:5484] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:55.495316 9382 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:55.503422 15356 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:55.503535 15356 ts_tablet_manager.cc:1367] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20240501 07:58:55.504007 15356 raft_consensus.cc:348] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.504140 15356 raft_consensus.cc:374] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.504171 15356 raft_consensus.cc:729] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.504281 15356 consensus_queue.cc:260] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.504577 15356 ts_tablet_manager.cc:1398] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.511885 15360 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:55.511978 15360 ts_tablet_manager.cc:1367] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.001s
I20240501 07:58:55.512444 15360 raft_consensus.cc:348] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.512549 15360 raft_consensus.cc:374] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.512576 15360 raft_consensus.cc:729] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.512684 15360 consensus_queue.cc:260] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.512923 15360 ts_tablet_manager.cc:1398] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.516878 15357 tablet_bootstrap.cc:492] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:55.517051 15357 ts_tablet_manager.cc:1367] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.001s
I20240501 07:58:55.517693 15357 raft_consensus.cc:348] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.521145 15358 tablet_bootstrap.cc:492] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:55.521247 15357 raft_consensus.cc:374] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.521330 15357 raft_consensus.cc:729] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.521464 15358 ts_tablet_manager.cc:1367] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.002s
I20240501 07:58:55.521641 15357 consensus_queue.cc:260] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.522233 15358 raft_consensus.cc:348] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.523736 7272 raft_consensus.cc:1203] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:55.523952 15357 ts_tablet_manager.cc:1398] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.007s user 0.000s sys 0.000s
I20240501 07:58:55.524753 15357 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:55.525774 15357 tablet_bootstrap.cc:654] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.526317 15358 raft_consensus.cc:374] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.526391 15358 raft_consensus.cc:729] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.526556 15358 consensus_queue.cc:260] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.527132 15357 tablet_bootstrap.cc:492] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:55.527257 15357 ts_tablet_manager.cc:1367] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:58:55.527480 15358 ts_tablet_manager.cc:1398] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:58:55.528107 15357 raft_consensus.cc:348] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.528270 15357 raft_consensus.cc:374] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.528357 15357 raft_consensus.cc:729] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.528504 15357 consensus_queue.cc:260] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.528867 15357 ts_tablet_manager.cc:1398] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.532374 15357 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:55.533886 15357 tablet_bootstrap.cc:654] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.540339 15357 tablet_bootstrap.cc:492] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:55.540446 15357 ts_tablet_manager.cc:1367] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:55.540839 15357 raft_consensus.cc:348] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.540957 15357 raft_consensus.cc:374] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.540997 15357 raft_consensus.cc:729] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.541110 15357 consensus_queue.cc:260] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.541402 15357 ts_tablet_manager.cc:1398] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.552170 6876 raft_consensus.cc:1203] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:55.556977 15382 ts_tablet_manager.cc:902] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:55.557471 15382 tablet_copy_client.cc:308] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:58:55.557931 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet f334b66a84b94c3ba15004253b4cb714 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:50687)
I20240501 07:58:55.558027 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet f334b66a84b94c3ba15004253b4cb714 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:50687: session id = 9a3888360d4d4ace96b6b909c160bfc2-f334b66a84b94c3ba15004253b4cb714
I20240501 07:58:55.559031 7160 tablet_copy_source_session.cc:205] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.560277 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f334b66a84b94c3ba15004253b4cb714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.562958 15382 tablet_copy_client.cc:783] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.563195 15382 tablet_copy_client.cc:647] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.567842 15382 tablet_copy_client.cc:523] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.572018 15382 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:55.573980 9665 raft_consensus.cc:1203] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:55.577701 15021 raft_consensus.cc:482] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.577803 15021 raft_consensus.cc:504] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:55.578191 15021 leader_election.cc:288] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:55.578476 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:55.578627 9665 raft_consensus.cc:2379] T dadf963e650a45bd978cb5f0f027ad5a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc946354b5d04503a56122e59fc320be in current term 1: Already voted for candidate fa6964452854415a85ad72f8639768d6 in this term.
I20240501 07:58:55.578575 9402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadf963e650a45bd978cb5f0f027ad5a" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:55.578975 7076 leader_election.cc:302] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6
I20240501 07:58:55.579144 15021 raft_consensus.cc:3037] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.580143 15021 raft_consensus.cc:2726] T dadf963e650a45bd978cb5f0f027ad5a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:55.609759 15382 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.610198 15382 tablet_bootstrap.cc:492] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:58:55.610523 15382 ts_tablet_manager.cc:1367] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.004s
I20240501 07:58:55.611127 15382 raft_consensus.cc:348] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.611353 15382 raft_consensus.cc:729] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.611528 15382 consensus_queue.cc:260] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.611850 15382 ts_tablet_manager.cc:1398] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.612058 15424 ts_tablet_manager.cc:902] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:55.612522 15424 tablet_copy_client.cc:308] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:58:55.612953 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet 3c5e1de6fc394fcb98c64165d74ca317 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:44575)
I20240501 07:58:55.613080 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet 3c5e1de6fc394fcb98c64165d74ca317 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:44575: session id = 15ab96f6477b4291932409b80af07213-3c5e1de6fc394fcb98c64165d74ca317
I20240501 07:58:55.614135 9291 tablet_copy_source_session.cc:205] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.614614 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-f334b66a84b94c3ba15004253b4cb714 received from {username='slave'} at 127.0.99.130:50687
I20240501 07:58:55.614683 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-f334b66a84b94c3ba15004253b4cb714 on tablet f334b66a84b94c3ba15004253b4cb714 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:58:55.614993 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e1de6fc394fcb98c64165d74ca317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.616155 15038 raft_consensus.cc:482] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.616255 15038 raft_consensus.cc:504] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.616674 15038 leader_election.cc:288] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:55.616986 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f4efcd9e274be483079915323045ad" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:55.617106 7010 raft_consensus.cc:2454] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:55.617213 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f4efcd9e274be483079915323045ad" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:55.617377 7138 raft_consensus.cc:2454] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:55.617424 9339 leader_election.cc:302] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:55.617686 15038 raft_consensus.cc:2781] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.617746 15038 raft_consensus.cc:482] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.617781 15038 raft_consensus.cc:3037] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.618763 15038 raft_consensus.cc:504] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.619231 15038 leader_election.cc:288] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:55.619715 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f4efcd9e274be483079915323045ad" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:55.619822 7010 raft_consensus.cc:3037] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.620110 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f4efcd9e274be483079915323045ad" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:55.620203 7138 raft_consensus.cc:3037] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.620442 15424 tablet_copy_client.cc:783] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.620651 15424 tablet_copy_client.cc:647] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.620795 7010 raft_consensus.cc:2454] T 12f4efcd9e274be483079915323045ad P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:55.621241 7138 raft_consensus.cc:2454] T 12f4efcd9e274be483079915323045ad P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:55.621699 9337 leader_election.cc:302] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:55.621868 15038 raft_consensus.cc:2781] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.621960 15038 raft_consensus.cc:686] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:55.622092 15038 consensus_queue.cc:237] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.623832 15424 tablet_copy_client.cc:523] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.624296 6732 catalog_manager.cc:5484] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.625236 15424 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:55.634454 15424 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.634795 15424 tablet_bootstrap.cc:492] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:58:55.635022 15424 ts_tablet_manager.cc:1367] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240501 07:58:55.635509 15424 raft_consensus.cc:348] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.635708 15424 raft_consensus.cc:729] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.635929 15424 consensus_queue.cc:260] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.636229 15424 ts_tablet_manager.cc:1398] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.636859 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-3c5e1de6fc394fcb98c64165d74ca317 received from {username='slave'} at 127.0.99.132:44575
I20240501 07:58:55.636937 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session 15ab96f6477b4291932409b80af07213-3c5e1de6fc394fcb98c64165d74ca317 on tablet 3c5e1de6fc394fcb98c64165d74ca317 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:58:55.646140 15226 ts_tablet_manager.cc:902] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:55.646656 15226 tablet_copy_client.cc:308] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:58:55.647573 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet c684b44a722143c4a3c44caad7cd84be from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:56749)
I20240501 07:58:55.647679 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet c684b44a722143c4a3c44caad7cd84be from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:56749: session id = 49adb9e99822494196ed22044bfed81f-c684b44a722143c4a3c44caad7cd84be
I20240501 07:58:55.648770 9686 tablet_copy_source_session.cc:205] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.649608 15226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684b44a722143c4a3c44caad7cd84be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.651902 15226 tablet_copy_client.cc:783] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.652060 15226 tablet_copy_client.cc:647] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.654912 15226 tablet_copy_client.cc:523] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.656736 15226 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:55.660033 14116 raft_consensus.cc:482] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.660127 14116 raft_consensus.cc:504] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.660611 14116 leader_election.cc:288] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:55.660975 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59513da29ad42939fa2ee264abce9fb" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:55.661010 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59513da29ad42939fa2ee264abce9fb" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:55.661090 9271 raft_consensus.cc:2454] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:55.661139 7138 raft_consensus.cc:2454] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:55.661394 9470 leader_election.cc:302] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:55.661504 15226 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.661553 14116 raft_consensus.cc:2781] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.661615 14116 raft_consensus.cc:482] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.661661 14116 raft_consensus.cc:3037] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.662004 15226 tablet_bootstrap.cc:492] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:58:55.662170 15226 ts_tablet_manager.cc:1367] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 07:58:55.662701 14116 raft_consensus.cc:504] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.662695 15226 raft_consensus.cc:348] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.662871 15226 raft_consensus.cc:729] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.663038 14116 leader_election.cc:288] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:55.663044 15226 consensus_queue.cc:260] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.663681 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59513da29ad42939fa2ee264abce9fb" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:55.663789 9271 raft_consensus.cc:3037] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.664165 15226 ts_tablet_manager.cc:1398] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.664793 9271 raft_consensus.cc:2454] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:55.665445 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-c684b44a722143c4a3c44caad7cd84be received from {username='slave'} at 127.0.99.133:56749
I20240501 07:58:55.665530 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 49adb9e99822494196ed22044bfed81f-c684b44a722143c4a3c44caad7cd84be on tablet c684b44a722143c4a3c44caad7cd84be with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:58:55.666009 14116 raft_consensus.cc:482] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.666082 14116 raft_consensus.cc:504] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.666383 14116 leader_election.cc:288] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:55.667474 9470 leader_election.cc:302] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:55.667835 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59513da29ad42939fa2ee264abce9fb" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:55.667917 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:55.668059 7140 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:55.667955 7138 raft_consensus.cc:3037] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.669221 7138 raft_consensus.cc:2454] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:55.670259 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:55.670393 7010 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:55.670691 9471 leader_election.cc:302] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:55.670857 15376 raft_consensus.cc:2781] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.671047 15376 raft_consensus.cc:686] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:55.671180 15445 raft_consensus.cc:2781] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.671236 15445 raft_consensus.cc:482] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.671326 15445 raft_consensus.cc:3037] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.671237 15376 consensus_queue.cc:237] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:55.673899 6735 catalog_manager.cc:5484] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:55.677031 15445 raft_consensus.cc:504] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.677325 15445 leader_election.cc:288] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:55.677675 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:55.677728 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:55.677806 7010 raft_consensus.cc:3037] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.677839 7138 raft_consensus.cc:3037] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.678650 7138 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:55.678715 7010 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:55.678936 9469 leader_election.cc:302] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:55.679296 15445 raft_consensus.cc:2781] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.679348 15447 ts_tablet_manager.cc:902] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:55.679458 15445 raft_consensus.cc:686] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:55.679596 15445 consensus_queue.cc:237] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:55.679824 15447 tablet_copy_client.cc:308] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:58:55.680234 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet d3333244e06a4ad8bd3ecc73cb356c90 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:45485)
I20240501 07:58:55.680336 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet d3333244e06a4ad8bd3ecc73cb356c90 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:45485: session id = 7fa3a88c25964445975bb47e0feed460-d3333244e06a4ad8bd3ecc73cb356c90
I20240501 07:58:55.681581 6735 catalog_manager.cc:5484] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:55.702730 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.703948 9414 tablet_copy_source_session.cc:205] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.705703 15447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3333244e06a4ad8bd3ecc73cb356c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.708030 15447 tablet_copy_client.cc:783] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.708187 15447 tablet_copy_client.cc:647] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.709224 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.711994 15447 tablet_copy_client.cc:523] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.712535 7138 raft_consensus.cc:1203] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:55.713306 15447 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:55.716013 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.718747 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.718894 15447 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.722215 9507 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:55.723265 9382 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:55.726161 9238 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.735308 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.743510 15447 tablet_bootstrap.cc:492] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:58:55.743695 15447 ts_tablet_manager.cc:1367] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20240501 07:58:55.744127 15447 raft_consensus.cc:348] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:55.744241 15447 raft_consensus.cc:729] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.744350 15447 consensus_queue.cc:260] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:55.744609 15447 ts_tablet_manager.cc:1398] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:55.745635 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-d3333244e06a4ad8bd3ecc73cb356c90 received from {username='slave'} at 127.0.99.136:45485
I20240501 07:58:55.745725 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-d3333244e06a4ad8bd3ecc73cb356c90 on tablet d3333244e06a4ad8bd3ecc73cb356c90 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:55.746925 9271 raft_consensus.cc:1203] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:55.759636 15466 ts_tablet_manager.cc:902] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:55.759896 15424 ts_tablet_manager.cc:902] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:55.760155 15466 tablet_copy_client.cc:308] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:58:55.760278 15424 tablet_copy_client.cc:308] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:58:55.760777 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 748b249371a4464f88aeb0c4c8558998 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:37717)
I20240501 07:58:55.760879 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 748b249371a4464f88aeb0c4c8558998 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:37717: session id = 15ab96f6477b4291932409b80af07213-748b249371a4464f88aeb0c4c8558998
I20240501 07:58:55.761739 9686 tablet_copy_source_session.cc:205] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.762132 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet 6a550051551640dcab9733948bb0966f from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:49879)
I20240501 07:58:55.762220 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet 6a550051551640dcab9733948bb0966f from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:49879: session id = fa6964452854415a85ad72f8639768d6-6a550051551640dcab9733948bb0966f
I20240501 07:58:55.763216 9291 tablet_copy_source_session.cc:205] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.763484 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748b249371a4464f88aeb0c4c8558998. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.765544 15424 tablet_copy_client.cc:783] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.765713 15424 tablet_copy_client.cc:647] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.767063 15424 tablet_copy_client.cc:523] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.769239 15424 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:55.774094 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a550051551640dcab9733948bb0966f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.776934 15466 tablet_copy_client.cc:783] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.777205 15466 tablet_copy_client.cc:647] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.778199 15466 tablet_copy_client.cc:523] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.779384 15466 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:55.786765 15424 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.787040 15424 tablet_bootstrap.cc:492] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:58:55.787200 15424 ts_tablet_manager.cc:1367] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240501 07:58:55.787647 15424 raft_consensus.cc:348] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.787766 15424 raft_consensus.cc:729] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.788022 15424 consensus_queue.cc:260] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:55.788394 15424 ts_tablet_manager.cc:1398] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:55.788951 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-748b249371a4464f88aeb0c4c8558998 received from {username='slave'} at 127.0.99.132:37717
I20240501 07:58:55.789026 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 15ab96f6477b4291932409b80af07213-748b249371a4464f88aeb0c4c8558998 on tablet 748b249371a4464f88aeb0c4c8558998 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:58:55.796527 15466 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.796809 15466 tablet_bootstrap.cc:492] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:58:55.796986 15466 ts_tablet_manager.cc:1367] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240501 07:58:55.797559 15466 raft_consensus.cc:348] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:55.797667 15466 raft_consensus.cc:729] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.797755 15466 consensus_queue.cc:260] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:55.798036 15466 ts_tablet_manager.cc:1398] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.798502 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-6a550051551640dcab9733948bb0966f received from {username='slave'} at 127.0.99.134:49879
I20240501 07:58:55.798584 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session fa6964452854415a85ad72f8639768d6-6a550051551640dcab9733948bb0966f on tablet 6a550051551640dcab9733948bb0966f with peer fa6964452854415a85ad72f8639768d6
I20240501 07:58:55.812628 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:55.817512 9271 raft_consensus.cc:1261] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.817996 15445 consensus_queue.cc:1035] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.818161 7138 raft_consensus.cc:1261] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.818532 15445 consensus_queue.cc:1035] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.822052 6735 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]). Expected version 1 got 0
I20240501 07:58:55.823133 6735 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]). Expected version 1 got 0
I20240501 07:58:55.826205 15494 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:55.826704 15494 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:55.828235 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]). Expected version 1 got 0
I20240501 07:58:55.829936 15496 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:55.829216 15493 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:55.830250 15493 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:55.830441 15496 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:55.834367 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d] alter complete (version 1)
I20240501 07:58:55.834619 15491 mvcc.cc:205] Tried to move back new op lower bound from 7022798175503081472 to 7022798174911008768. Current Snapshot: MvccSnapshot[applied={T|T < 7022798175503081472}]
I20240501 07:58:55.836275 15495 ts_tablet_manager.cc:902] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:55.836767 15495 tablet_copy_client.cc:308] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:58:55.837209 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet 79ab7f97eb2e439f8c974636b5791824 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:34851)
I20240501 07:58:55.837327 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet 79ab7f97eb2e439f8c974636b5791824 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:34851: session id = 1a9029bee6a144938f68b363e4cca162-79ab7f97eb2e439f8c974636b5791824
I20240501 07:58:55.838182 9414 tablet_copy_source_session.cc:205] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:55.838941 15495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ab7f97eb2e439f8c974636b5791824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.841039 15495 tablet_copy_client.cc:783] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:55.841187 15495 tablet_copy_client.cc:647] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:55.842151 15495 tablet_copy_client.cc:523] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:55.843222 15495 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:55.847368 15495 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:55.847616 15495 tablet_bootstrap.cc:492] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:58:55.847781 15495 ts_tablet_manager.cc:1367] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:55.848177 15495 raft_consensus.cc:348] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:55.848276 15495 raft_consensus.cc:729] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.848421 15495 consensus_queue.cc:260] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:55.848707 15495 ts_tablet_manager.cc:1398] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.849296 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-79ab7f97eb2e439f8c974636b5791824 received from {username='slave'} at 127.0.99.129:34851
I20240501 07:58:55.849373 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-79ab7f97eb2e439f8c974636b5791824 on tablet 79ab7f97eb2e439f8c974636b5791824 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:55.919133 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.928857 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.929129 9238 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.930039 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:55.931139 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:55.931175 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:55.937069 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:55.945536 9507 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:55.945868 15493 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.946210 15493 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:55.950702 15496 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.951089 15494 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:55.954689 9382 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:55.951357 15494 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:55.950990 15496 tablet.cc:1620] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:55.963433 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d] alter complete (version 2)
I20240501 07:58:55.967708 7010 raft_consensus.cc:1203] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:58:55.969098 15038 consensus_queue.cc:1035] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.979151 15181 consensus_queue.cc:1035] T dadf963e650a45bd978cb5f0f027ad5a P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.007889 15181 consensus_queue.cc:1035] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.011402 15181 consensus_queue.cc:1035] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.071355 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:56.077487 15493 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.077976 15494 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.082650 15496 tablet.cc:1722] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.085122 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d] alter complete (version 3)
I20240501 07:58:56.100502 9266 raft_consensus.cc:1203] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:56.104172 15376 consensus_queue.cc:1035] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.108412 14116 consensus_queue.cc:1035] T 76cd704e855744f2b1e95d125f282e5f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.132112 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.138012 9238 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:56.139799 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:56.142405 9382 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:56.144863 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.146358 9402 raft_consensus.cc:1203] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:56.149940 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.156365 9645 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.160511 9665 raft_consensus.cc:1203] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:56.173153 9507 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:56.186776 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:56.186926 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:56.189095 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet b59513da29ad42939fa2ee264abce9fb
I20240501 07:58:56.189774 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet b59513da29ad42939fa2ee264abce9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:56 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:56.189941 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet b59513da29ad42939fa2ee264abce9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:56 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:56.191084 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet b59513da29ad42939fa2ee264abce9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:56 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:56.196327 15605 tablet_replica.cc:330] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:56.196497 15605 raft_consensus.cc:2229] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:56.196748 15605 raft_consensus.cc:2258] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:56.197679 15605 ts_tablet_manager.cc:1875] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:56.199622 15607 tablet_replica.cc:330] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:56.199775 15607 raft_consensus.cc:2229] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:56.199898 15607 raft_consensus.cc:2258] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:56.200114 15606 tablet_replica.cc:330] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:56.200263 15606 raft_consensus.cc:2229] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:56.200438 15606 raft_consensus.cc:2258] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:56.200899 15607 ts_tablet_manager.cc:1875] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:56.201964 15605 ts_tablet_manager.cc:1888] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:56.202037 15605 log.cc:1196] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b59513da29ad42939fa2ee264abce9fb
I20240501 07:58:56.202299 15605 ts_tablet_manager.cc:1909] T b59513da29ad42939fa2ee264abce9fb P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:56.202908 15607 ts_tablet_manager.cc:1888] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:56.202975 15607 log.cc:1196] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b59513da29ad42939fa2ee264abce9fb
I20240501 07:58:56.203272 15607 ts_tablet_manager.cc:1909] T b59513da29ad42939fa2ee264abce9fb P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:56.203857 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table_new_name [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]) successfully deleted
I20240501 07:58:56.204139 15606 ts_tablet_manager.cc:1875] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:56.205787 15606 ts_tablet_manager.cc:1888] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:56.205854 15606 log.cc:1196] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b59513da29ad42939fa2ee264abce9fb
I20240501 07:58:56.206063 15606 ts_tablet_manager.cc:1909] T b59513da29ad42939fa2ee264abce9fb P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:56.206590 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table_new_name [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]) successfully deleted
I20240501 07:58:56.206921 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet b59513da29ad42939fa2ee264abce9fb (table rebalancer_dynamic_table_new_name [id=ed6d2dc97e25410bb6bb4cd8b2cbcf8d]) successfully deleted
I20240501 07:58:56.210083 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:56.215973 7272 raft_consensus.cc:1203] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:56.217494 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb409ae7134b07b8a809a9d1aabdd4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.217774 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb409ae7134b07b8a809a9d1aabdd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.218134 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 6b07121b184343038c727463fea20ffd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.218385 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b07121b184343038c727463fea20ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.220247 9646 tablet_service.cc:1467] Processing CreateTablet for tablet e90ac95b538e419ea78e36e54ad8b4ac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.220474 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90ac95b538e419ea78e36e54ad8b4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.222900 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 6b07121b184343038c727463fea20ffd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.222973 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb409ae7134b07b8a809a9d1aabdd4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.223212 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb409ae7134b07b8a809a9d1aabdd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.223212 9251 tablet_service.cc:1467] Processing CreateTablet for tablet e90ac95b538e419ea78e36e54ad8b4ac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.223439 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90ac95b538e419ea78e36e54ad8b4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.225662 9515 tablet_service.cc:1467] Processing CreateTablet for tablet e90ac95b538e419ea78e36e54ad8b4ac (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.225888 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90ac95b538e419ea78e36e54ad8b4ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.228760 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b07121b184343038c727463fea20ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.232059 15609 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:56.233222 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb409ae7134b07b8a809a9d1aabdd4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.233273 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 6b07121b184343038c727463fea20ffd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=521a8fde4d4a492eab395b34a2d27de8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.233464 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb409ae7134b07b8a809a9d1aabdd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.233469 15610 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:56.233821 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b07121b184343038c727463fea20ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.234452 15610 tablet_bootstrap.cc:654] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.235698 15608 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:56.236191 15609 tablet_bootstrap.cc:654] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.236891 15608 tablet_bootstrap.cc:654] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.238309 15610 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:56.238408 15610 ts_tablet_manager.cc:1367] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:56.238976 15610 raft_consensus.cc:348] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.239166 15610 raft_consensus.cc:374] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.239239 15610 raft_consensus.cc:729] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.239446 15610 consensus_queue.cc:260] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.239734 15610 ts_tablet_manager.cc:1398] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.241061 15610 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:56.241283 6876 raft_consensus.cc:1203] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:56.243530 15610 tablet_bootstrap.cc:654] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.244973 15608 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:56.245008 15609 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:56.245072 15608 ts_tablet_manager.cc:1367] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:56.245101 15609 ts_tablet_manager.cc:1367] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:58:56.245620 15609 raft_consensus.cc:348] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.245724 15609 raft_consensus.cc:374] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.245759 15609 raft_consensus.cc:729] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.245880 15613 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:56.245870 15609 consensus_queue.cc:260] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.246158 15609 ts_tablet_manager.cc:1398] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.246344 15609 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:56.246836 15613 tablet_bootstrap.cc:654] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.247287 15609 tablet_bootstrap.cc:654] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.245874 15608 raft_consensus.cc:348] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.247814 15608 raft_consensus.cc:374] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.247888 15608 raft_consensus.cc:729] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.248049 15608 consensus_queue.cc:260] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.250162 15608 ts_tablet_manager.cc:1398] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:56.250387 15608 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:56.253379 15608 tablet_bootstrap.cc:654] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.254472 15608 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:56.254559 15608 ts_tablet_manager.cc:1367] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:56.254920 15608 raft_consensus.cc:348] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.255031 15608 raft_consensus.cc:374] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.255090 15608 raft_consensus.cc:729] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.255465 15608 consensus_queue.cc:260] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.256242 15610 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:56.256323 15610 ts_tablet_manager.cc:1367] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:56.256580 15608 ts_tablet_manager.cc:1398] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:56.256700 15610 raft_consensus.cc:348] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.256801 15610 raft_consensus.cc:374] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.256834 15610 raft_consensus.cc:729] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.256778 15608 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:56.256948 15610 consensus_queue.cc:260] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.257277 15610 ts_tablet_manager.cc:1398] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.257562 15613 tablet_bootstrap.cc:492] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:56.257946 15613 ts_tablet_manager.cc:1367] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:56.258028 15609 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:56.258103 15609 ts_tablet_manager.cc:1367] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 07:58:56.258520 15609 raft_consensus.cc:348] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.258517 15613 raft_consensus.cc:348] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.258612 15609 raft_consensus.cc:374] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.258612 15613 raft_consensus.cc:374] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.258644 15609 raft_consensus.cc:729] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.258644 15613 raft_consensus.cc:729] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.258765 15609 consensus_queue.cc:260] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.259049 15609 ts_tablet_manager.cc:1398] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.258941 15613 consensus_queue.cc:260] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.259498 15613 ts_tablet_manager.cc:1398] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.259570 15608 tablet_bootstrap.cc:654] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.259748 15613 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:56.260545 15613 tablet_bootstrap.cc:654] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.261267 15608 tablet_bootstrap.cc:492] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:56.261543 15613 tablet_bootstrap.cc:492] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:56.261829 15613 ts_tablet_manager.cc:1367] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:56.262176 15613 raft_consensus.cc:348] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.262253 15613 raft_consensus.cc:374] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.262370 15613 raft_consensus.cc:729] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.262517 15613 consensus_queue.cc:260] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.262853 15613 ts_tablet_manager.cc:1398] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.262924 15608 ts_tablet_manager.cc:1367] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 07:58:56.264086 15608 raft_consensus.cc:348] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.264258 15608 raft_consensus.cc:374] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.264338 15608 raft_consensus.cc:729] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.266515 15608 consensus_queue.cc:260] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.266806 15608 ts_tablet_manager.cc:1398] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:56.284098 7272 raft_consensus.cc:1203] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:58:56.290833 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:56.296499 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]), partition=
I20240501 07:58:56.296635 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]), partition=
I20240501 07:58:56.296789 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930da16f02ad466f9f4ee2c46e40bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.296864 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930da16f02ad466f9f4ee2c46e40bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.299249 15638 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:56.299245 15639 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:56.299762 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]), partition=
I20240501 07:58:56.299997 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930da16f02ad466f9f4ee2c46e40bfaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.300724 15638 tablet_bootstrap.cc:654] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.300824 15639 tablet_bootstrap.cc:654] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.301851 15638 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:56.301941 15638 ts_tablet_manager.cc:1367] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:56.302336 15610 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:56.302438 15638 raft_consensus.cc:348] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.302551 15638 raft_consensus.cc:374] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.302579 15638 raft_consensus.cc:729] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.302699 15638 consensus_queue.cc:260] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.302955 15638 ts_tablet_manager.cc:1398] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.303229 15610 tablet_bootstrap.cc:654] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.303637 15206 raft_consensus.cc:482] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.303723 15206 raft_consensus.cc:504] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.304106 15206 leader_election.cc:288] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:56.304484 15610 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:56.304569 15610 ts_tablet_manager.cc:1367] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:56.304553 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb409ae7134b07b8a809a9d1aabdd4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:56.304684 9535 raft_consensus.cc:2454] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:56.304997 9602 leader_election.cc:302] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:56.305011 15610 raft_consensus.cc:348] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.305094 15610 raft_consensus.cc:374] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.305135 15610 raft_consensus.cc:729] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.305225 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb409ae7134b07b8a809a9d1aabdd4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:56.305253 15610 consensus_queue.cc:260] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.305359 7010 raft_consensus.cc:2454] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:56.305562 15610 ts_tablet_manager.cc:1398] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.307305 15206 raft_consensus.cc:2781] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.307376 15206 raft_consensus.cc:482] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.307421 15206 raft_consensus.cc:3037] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.308452 15206 raft_consensus.cc:504] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.308701 15206 leader_election.cc:288] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:56.309033 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb409ae7134b07b8a809a9d1aabdd4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:56.309147 9535 raft_consensus.cc:3037] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.309350 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb409ae7134b07b8a809a9d1aabdd4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:56.309438 7010 raft_consensus.cc:3037] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.310036 15639 tablet_bootstrap.cc:492] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:56.310163 15639 ts_tablet_manager.cc:1367] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:56.310200 9535 raft_consensus.cc:2454] T 17bb409ae7134b07b8a809a9d1aabdd4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:56.310326 7010 raft_consensus.cc:2454] T 17bb409ae7134b07b8a809a9d1aabdd4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:56.310503 9602 leader_election.cc:302] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:56.310663 15206 raft_consensus.cc:2781] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.310781 15206 raft_consensus.cc:686] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:56.310917 15206 consensus_queue.cc:237] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.311746 15639 raft_consensus.cc:348] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.311849 15639 raft_consensus.cc:374] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.311883 15639 raft_consensus.cc:729] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.312039 15639 consensus_queue.cc:260] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.312376 15639 ts_tablet_manager.cc:1398] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:56.313205 6737 catalog_manager.cc:5484] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.382261 15611 raft_consensus.cc:482] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.382381 15611 raft_consensus.cc:504] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.382953 15611 leader_election.cc:288] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:56.383220 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b07121b184343038c727463fea20ffd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:56.383344 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b07121b184343038c727463fea20ffd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:56.383389 9665 raft_consensus.cc:2454] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:56.383463 7010 raft_consensus.cc:2454] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:56.383635 9206 leader_election.cc:302] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:56.383857 15611 raft_consensus.cc:2781] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.383903 15611 raft_consensus.cc:482] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.383949 15611 raft_consensus.cc:3037] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.384797 15611 raft_consensus.cc:504] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.385166 15611 leader_election.cc:288] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:56.385493 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b07121b184343038c727463fea20ffd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:56.385592 9665 raft_consensus.cc:3037] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.385588 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b07121b184343038c727463fea20ffd" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:56.385697 7010 raft_consensus.cc:3037] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.386559 7010 raft_consensus.cc:2454] T 6b07121b184343038c727463fea20ffd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:56.386574 9665 raft_consensus.cc:2454] T 6b07121b184343038c727463fea20ffd P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:56.386919 9206 leader_election.cc:302] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:58:56.387086 15611 raft_consensus.cc:2781] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.387166 15611 raft_consensus.cc:686] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:56.387305 15611 consensus_queue.cc:237] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.390216 6731 catalog_manager.cc:5484] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.487018 15611 raft_consensus.cc:482] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.487146 15611 raft_consensus.cc:504] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.487519 15611 leader_election.cc:288] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:56.487848 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90ac95b538e419ea78e36e54ad8b4ac" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:56.487879 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90ac95b538e419ea78e36e54ad8b4ac" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:56.487967 9535 raft_consensus.cc:2454] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:56.488001 9665 raft_consensus.cc:2454] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:56.488204 9207 leader_election.cc:302] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:56.488360 15611 raft_consensus.cc:2781] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.488418 15611 raft_consensus.cc:482] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.488466 15611 raft_consensus.cc:3037] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.489439 15611 raft_consensus.cc:504] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.490034 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90ac95b538e419ea78e36e54ad8b4ac" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:56.490150 9535 raft_consensus.cc:3037] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.490289 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90ac95b538e419ea78e36e54ad8b4ac" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:56.490386 9665 raft_consensus.cc:3037] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.491281 9535 raft_consensus.cc:2454] T e90ac95b538e419ea78e36e54ad8b4ac P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:56.491340 9665 raft_consensus.cc:2454] T e90ac95b538e419ea78e36e54ad8b4ac P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:56.491622 9207 leader_election.cc:302] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:56.491833 15278 raft_consensus.cc:2781] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.492020 15278 raft_consensus.cc:686] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:56.492179 15278 consensus_queue.cc:237] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:56.493204 15611 leader_election.cc:288] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:56.494683 6731 catalog_manager.cc:5484] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:56.527081 15220 raft_consensus.cc:482] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.527190 15220 raft_consensus.cc:504] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.527667 15220 leader_election.cc:288] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:56.527983 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930da16f02ad466f9f4ee2c46e40bfaf" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:56.528009 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930da16f02ad466f9f4ee2c46e40bfaf" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:56.528096 9403 raft_consensus.cc:2454] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:56.528170 9266 raft_consensus.cc:2454] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:56.528376 6814 leader_election.cc:302] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:56.528535 15220 raft_consensus.cc:2781] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.528592 15220 raft_consensus.cc:482] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.528633 15220 raft_consensus.cc:3037] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.529733 15220 raft_consensus.cc:504] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.530038 15220 leader_election.cc:288] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:56.530424 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930da16f02ad466f9f4ee2c46e40bfaf" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:56.530484 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930da16f02ad466f9f4ee2c46e40bfaf" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:56.530524 9403 raft_consensus.cc:3037] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.530578 9266 raft_consensus.cc:3037] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.531448 9403 raft_consensus.cc:2454] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:56.531711 6814 leader_election.cc:302] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:56.531878 15220 raft_consensus.cc:2781] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.531947 15220 raft_consensus.cc:686] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:56.532084 15220 consensus_queue.cc:237] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.534271 6730 catalog_manager.cc:5484] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:56.552111 9266 raft_consensus.cc:2454] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:56.567651 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.579269 9515 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:58:56.579897 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.580837 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.583854 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.595989 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.596194 9382 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:56.628809 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.644636 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:56.649187 9266 raft_consensus.cc:1261] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.649236 9403 raft_consensus.cc:1261] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.649653 14700 consensus_queue.cc:1035] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.649943 15220 consensus_queue.cc:1035] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.652776 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]). Expected version 1 got 0
I20240501 07:58:56.653527 6730 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]). Expected version 1 got 0
I20240501 07:58:56.654776 15691 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:56.655124 15691 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:56.656213 6730 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]). Expected version 1 got 0
I20240501 07:58:56.656365 15692 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:56.656457 15690 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:56.656672 15692 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:56.656728 15690 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:56.660240 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f]). Expected version 1 got 0
I20240501 07:58:56.662842 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f] alter complete (version 1)
I20240501 07:58:56.685388 9665 consensus_queue.cc:237] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:56.686594 9266 raft_consensus.cc:1261] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:56.686798 7010 raft_consensus.cc:1261] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:56.686954 14505 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:56.687155 15206 consensus_queue.cc:1035] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:56.689458 15206 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:56.690032 9266 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:56.692618 6730 catalog_manager.cc:5484] T c684b44a722143c4a3c44caad7cd84be P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:56.694447 7010 raft_consensus.cc:2932] T c684b44a722143c4a3c44caad7cd84be P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:56.695343 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet c684b44a722143c4a3c44caad7cd84be with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:56.695912 15716 tablet_replica.cc:330] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:56.696344 15716 raft_consensus.cc:2229] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:56.696619 15716 raft_consensus.cc:2258] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:56.697172 15716 ts_tablet_manager.cc:1875] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:56.704272 15716 ts_tablet_manager.cc:1888] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:56.704507 15716 log.cc:1196] T c684b44a722143c4a3c44caad7cd84be P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c684b44a722143c4a3c44caad7cd84be
I20240501 07:58:56.705088 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet c684b44a722143c4a3c44caad7cd84be (table rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]) successfully deleted
I20240501 07:58:56.765897 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:56.771937 15690 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:56.772332 15690 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:58:56.774421 14505 consensus_queue.cc:1035] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.779639 15206 consensus_queue.cc:1035] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.781144 15691 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:56.781507 15691 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:56.781736 15692 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:56.782052 15692 tablet.cc:1620] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:56.791921 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=bbbbb864e4e34b008eefd7bcd5ed446f] alter complete (version 2)
I20240501 07:58:56.820974 7251 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.824807 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.829457 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.835809 9515 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:58:56.843538 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.850164 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:56.856726 9382 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:56.879891 15611 consensus_queue.cc:1035] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.892112 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:56.894796 15611 consensus_queue.cc:1035] T 6b07121b184343038c727463fea20ffd P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.898460 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:56.905835 15690 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.906527 15692 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.912551 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=bbbbb864e4e34b008eefd7bcd5ed446f] alter complete (version 3)
I20240501 07:58:56.914068 15691 tablet.cc:1722] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:56.938047 7138 consensus_queue.cc:237] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:56.939365 7009 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:56.939699 9535 raft_consensus.cc:1261] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:56.939726 15684 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:56.940124 15684 consensus_queue.cc:1035] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:56.941649 15684 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.942035 7009 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.942060 9535 raft_consensus.cc:2932] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.944733 6736 catalog_manager.cc:5484] T f334b66a84b94c3ba15004253b4cb714 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 6 to 7, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:56.947993 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet f334b66a84b94c3ba15004253b4cb714 with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:56.949829 15796 tablet_replica.cc:330] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:56.950016 15796 raft_consensus.cc:2229] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:56.950193 15796 raft_consensus.cc:2258] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:56.950778 15796 ts_tablet_manager.cc:1875] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:56.953073 15796 ts_tablet_manager.cc:1888] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:58:56.953208 15796 log.cc:1196] T f334b66a84b94c3ba15004253b4cb714 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f334b66a84b94c3ba15004253b4cb714
I20240501 07:58:56.953853 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet f334b66a84b94c3ba15004253b4cb714 (table rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]) successfully deleted
I20240501 07:58:56.977520 15611 consensus_queue.cc:1035] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.980743 15611 consensus_queue.cc:1035] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.014741 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:57.014879 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:57.017323 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 930da16f02ad466f9f4ee2c46e40bfaf
I20240501 07:58:57.017858 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:57.018023 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:58:57.018065 15716 tablet_replica.cc:330] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:57.018247 15716 raft_consensus.cc:2229] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.018378 15716 raft_consensus.cc:2258] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.018579 15796 tablet_replica.cc:330] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:58:57.018743 15796 raft_consensus.cc:2229] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:57.018945 15796 raft_consensus.cc:2258] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.019052 15716 ts_tablet_manager.cc:1875] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.019116 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 930da16f02ad466f9f4ee2c46e40bfaf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:57.021425 15716 ts_tablet_manager.cc:1888] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.021502 15716 log.cc:1196] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/930da16f02ad466f9f4ee2c46e40bfaf
I20240501 07:58:57.021528 15796 ts_tablet_manager.cc:1875] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.021792 15716 ts_tablet_manager.cc:1909] T 930da16f02ad466f9f4ee2c46e40bfaf P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:57.022482 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table_new_name [id=bbbbb864e4e34b008eefd7bcd5ed446f]) successfully deleted
I20240501 07:58:57.023746 15796 ts_tablet_manager.cc:1888] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.023811 15796 log.cc:1196] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/930da16f02ad466f9f4ee2c46e40bfaf
I20240501 07:58:57.024072 15796 ts_tablet_manager.cc:1909] T 930da16f02ad466f9f4ee2c46e40bfaf P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:58:57.027846 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table_new_name [id=bbbbb864e4e34b008eefd7bcd5ed446f]) successfully deleted
I20240501 07:58:57.026575 15835 tablet_replica.cc:330] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:57.030272 15835 raft_consensus.cc:2229] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.030444 15835 raft_consensus.cc:2258] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.031114 15835 ts_tablet_manager.cc:1875] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.034235 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:57.035948 15835 ts_tablet_manager.cc:1888] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.036079 15835 log.cc:1196] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/930da16f02ad466f9f4ee2c46e40bfaf
I20240501 07:58:57.036439 15835 ts_tablet_manager.cc:1909] T 930da16f02ad466f9f4ee2c46e40bfaf P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:57.039916 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 930da16f02ad466f9f4ee2c46e40bfaf (table rebalancer_dynamic_table_new_name [id=bbbbb864e4e34b008eefd7bcd5ed446f]) successfully deleted
I20240501 07:58:57.045018 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f560797734bc458ea6d9c538d8bacf7d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.045346 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f560797734bc458ea6d9c538d8bacf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.045708 7250 tablet_service.cc:1467] Processing CreateTablet for tablet d0ed3a7796e548e781898ef10c2601bb (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.045708 9515 tablet_service.cc:1467] Processing CreateTablet for tablet f560797734bc458ea6d9c538d8bacf7d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.045957 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f560797734bc458ea6d9c538d8bacf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.045960 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ed3a7796e548e781898ef10c2601bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.047479 15836 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:57.047658 7121 tablet_service.cc:1467] Processing CreateTablet for tablet d0ed3a7796e548e781898ef10c2601bb (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.047906 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ed3a7796e548e781898ef10c2601bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.048431 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 3e10114b061d453c88cee34494da19d5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.048575 15836 tablet_bootstrap.cc:654] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.048661 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e10114b061d453c88cee34494da19d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.048627 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 3e10114b061d453c88cee34494da19d5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.048892 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e10114b061d453c88cee34494da19d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.050127 15839 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:57.051312 9646 tablet_service.cc:1467] Processing CreateTablet for tablet d0ed3a7796e548e781898ef10c2601bb (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.051582 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ed3a7796e548e781898ef10c2601bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.051600 15839 tablet_bootstrap.cc:654] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.052134 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 3e10114b061d453c88cee34494da19d5 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.052425 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e10114b061d453c88cee34494da19d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.053361 15839 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:57.053438 15839 ts_tablet_manager.cc:1367] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.053999 9382 tablet_service.cc:1467] Processing CreateTablet for tablet f560797734bc458ea6d9c538d8bacf7d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a103c0e5246948cf999d878868d8120a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.054245 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f560797734bc458ea6d9c538d8bacf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.054704 15839 raft_consensus.cc:348] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.054811 15839 raft_consensus.cc:374] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.054842 15839 raft_consensus.cc:729] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.054962 15839 consensus_queue.cc:260] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.055333 15839 ts_tablet_manager.cc:1398] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:57.059576 15843 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:57.060644 15843 tablet_bootstrap.cc:654] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.060925 15841 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:57.062088 15841 tablet_bootstrap.cc:654] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.062326 15836 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:57.062441 15836 ts_tablet_manager.cc:1367] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:57.063171 15836 raft_consensus.cc:348] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.063289 15836 raft_consensus.cc:374] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.063324 15836 raft_consensus.cc:729] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.063536 15843 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:57.063504 15836 consensus_queue.cc:260] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.063613 15843 ts_tablet_manager.cc:1367] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:57.063793 15836 ts_tablet_manager.cc:1398] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.063962 15836 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:57.064039 15843 raft_consensus.cc:348] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.064169 15843 raft_consensus.cc:374] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.064200 15843 raft_consensus.cc:729] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.064498 15843 consensus_queue.cc:260] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.065536 15836 tablet_bootstrap.cc:654] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.065567 15843 ts_tablet_manager.cc:1398] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:57.066515 15841 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:57.066601 15841 ts_tablet_manager.cc:1367] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:57.067137 15841 raft_consensus.cc:348] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.067260 15841 raft_consensus.cc:374] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.067296 15841 raft_consensus.cc:729] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.067415 15841 consensus_queue.cc:260] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.067708 15841 ts_tablet_manager.cc:1398] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.070952 15836 tablet_bootstrap.cc:492] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:57.071096 15836 ts_tablet_manager.cc:1367] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:57.071578 15836 raft_consensus.cc:348] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.071844 15836 raft_consensus.cc:374] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.072057 15836 raft_consensus.cc:729] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.072124 15842 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:57.072212 15836 consensus_queue.cc:260] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.073057 15837 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:57.073372 15836 ts_tablet_manager.cc:1398] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:57.073591 15836 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:57.074661 15836 tablet_bootstrap.cc:654] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.080909 15842 tablet_bootstrap.cc:654] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.082568 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.087255 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.087328 15837 tablet_bootstrap.cc:654] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.094704 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.099145 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.103528 15842 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:57.103688 15842 ts_tablet_manager.cc:1367] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240501 07:58:57.104379 15842 raft_consensus.cc:348] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.104568 15842 raft_consensus.cc:374] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.104640 15842 raft_consensus.cc:729] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.104838 15842 consensus_queue.cc:260] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.105247 15842 ts_tablet_manager.cc:1398] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.106197 7121 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:57.106675 15837 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:57.106783 15837 ts_tablet_manager.cc:1367] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240501 07:58:57.107470 15837 raft_consensus.cc:348] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.107575 15837 raft_consensus.cc:374] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.107602 15837 raft_consensus.cc:729] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.107671 15842 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:57.107719 15837 consensus_queue.cc:260] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.107983 15837 ts_tablet_manager.cc:1398] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.111663 15842 tablet_bootstrap.cc:654] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.112736 15464 raft_consensus.cc:482] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.112828 15464 raft_consensus.cc:504] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.115787 15464 leader_election.cc:288] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:57.116199 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ed3a7796e548e781898ef10c2601bb" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:57.116340 9666 raft_consensus.cc:2454] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:57.116765 15445 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.116847 15445 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.116947 7076 leader_election.cc:302] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:57.117240 15445 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:57.117359 15684 raft_consensus.cc:2781] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.117421 15684 raft_consensus.cc:482] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.117465 15684 raft_consensus.cc:3037] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.118547 15684 raft_consensus.cc:504] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.118880 15684 leader_election.cc:288] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:57.119252 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ed3a7796e548e781898ef10c2601bb" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:57.119364 9666 raft_consensus.cc:3037] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.119341 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:57.119463 7270 raft_consensus.cc:2454] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:57.119822 9469 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:57.119966 15445 raft_consensus.cc:2781] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.120002 15445 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.120035 15445 raft_consensus.cc:3037] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.120317 9666 raft_consensus.cc:2454] T d0ed3a7796e548e781898ef10c2601bb P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:57.120589 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:57.120922 15445 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.120952 7076 leader_election.cc:302] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:57.121235 15445 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:57.121325 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ed3a7796e548e781898ef10c2601bb" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:57.121441 7270 raft_consensus.cc:3037] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.121497 15464 raft_consensus.cc:2781] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.121572 15464 raft_consensus.cc:686] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:57.121692 15464 consensus_queue.cc:237] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.122460 7270 raft_consensus.cc:2454] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:57.122627 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:57.122704 7271 raft_consensus.cc:3037] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.123617 7271 raft_consensus.cc:2454] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:57.123914 9469 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:57.124064 15445 raft_consensus.cc:2781] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.123966 6729 catalog_manager.cc:5484] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:57.124120 15445 raft_consensus.cc:686] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:57.124228 15445 consensus_queue.cc:237] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
W20240501 07:58:57.120940 9470 leader_election.cc:341] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:57.125735 15836 tablet_bootstrap.cc:492] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:57.125849 15836 ts_tablet_manager.cc:1367] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.001s
I20240501 07:58:57.125873 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:57.126410 15836 raft_consensus.cc:348] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.126540 15836 raft_consensus.cc:374] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.126600 15836 raft_consensus.cc:729] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.126739 15836 consensus_queue.cc:260] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.127094 15836 ts_tablet_manager.cc:1398] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:57.127619 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ed3a7796e548e781898ef10c2601bb" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:57.127736 7272 raft_consensus.cc:2362] T d0ed3a7796e548e781898ef10c2601bb P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1. Re-sending same reply.
I20240501 07:58:57.128239 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:57.128270 6729 catalog_manager.cc:5484] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
W20240501 07:58:57.128680 9470 leader_election.cc:341] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:57.134667 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]), partition=
I20240501 07:58:57.134846 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]), partition=
I20240501 07:58:57.134948 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001a2aed4ca94c81a4f350da98ae8132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.135121 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001a2aed4ca94c81a4f350da98ae8132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.135684 9381 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:58:57.136991 15841 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:57.137180 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]), partition=
I20240501 07:58:57.137210 15839 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:57.137379 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001a2aed4ca94c81a4f350da98ae8132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.138078 15839 tablet_bootstrap.cc:654] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.138276 15841 tablet_bootstrap.cc:654] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.160494 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:57.164306 15839 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:57.164420 15839 ts_tablet_manager.cc:1367] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 07:58:57.164500 15842 tablet_bootstrap.cc:492] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:57.164585 15842 ts_tablet_manager.cc:1367] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
I20240501 07:58:57.164884 15839 raft_consensus.cc:348] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.165022 15839 raft_consensus.cc:374] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.165052 15839 raft_consensus.cc:729] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.165194 15839 consensus_queue.cc:260] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.165462 15839 ts_tablet_manager.cc:1398] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.165531 15842 raft_consensus.cc:348] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.165629 15842 raft_consensus.cc:374] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.165655 15842 raft_consensus.cc:729] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.165784 15842 consensus_queue.cc:260] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.166774 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:57.167119 15842 ts_tablet_manager.cc:1398] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:57.167320 15842 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:57.168305 15842 tablet_bootstrap.cc:654] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.168740 15841 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:57.168830 15841 ts_tablet_manager.cc:1367] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20240501 07:58:57.169230 15841 raft_consensus.cc:348] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.169319 15841 raft_consensus.cc:374] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.169346 15841 raft_consensus.cc:729] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.169452 15841 consensus_queue.cc:260] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.169692 15841 ts_tablet_manager.cc:1398] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.191748 15842 tablet_bootstrap.cc:492] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:57.191848 15842 ts_tablet_manager.cc:1367] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 07:58:57.192469 15842 raft_consensus.cc:348] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.192569 15842 raft_consensus.cc:374] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.192596 15842 raft_consensus.cc:729] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.192709 15842 consensus_queue.cc:260] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.192936 15842 ts_tablet_manager.cc:1398] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.200294 14505 raft_consensus.cc:482] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.200381 14505 raft_consensus.cc:504] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.200790 14505 leader_election.cc:288] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:57.201257 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001a2aed4ca94c81a4f350da98ae8132" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:57.201390 9403 raft_consensus.cc:2454] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:57.201577 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001a2aed4ca94c81a4f350da98ae8132" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:57.201700 7138 raft_consensus.cc:2454] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:58:57.202019 9601 leader_election.cc:302] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:57.202448 14505 raft_consensus.cc:2781] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.202518 14505 raft_consensus.cc:482] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.202554 14505 raft_consensus.cc:3037] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.203590 14505 raft_consensus.cc:504] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.203918 14505 leader_election.cc:288] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:57.204490 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001a2aed4ca94c81a4f350da98ae8132" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:57.204593 9403 raft_consensus.cc:3037] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.204573 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001a2aed4ca94c81a4f350da98ae8132" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:57.204671 7138 raft_consensus.cc:3037] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.205492 9403 raft_consensus.cc:2454] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:57.205617 7138 raft_consensus.cc:2454] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:58:57.205998 9601 leader_election.cc:302] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:57.206569 14505 raft_consensus.cc:2781] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.206640 14505 raft_consensus.cc:686] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:58:57.206748 14505 consensus_queue.cc:237] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.209216 6733 catalog_manager.cc:5484] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:57.217381 15181 raft_consensus.cc:482] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.217471 15181 raft_consensus.cc:504] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.217882 15181 leader_election.cc:288] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:57.218195 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e10114b061d453c88cee34494da19d5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:57.218282 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e10114b061d453c88cee34494da19d5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:57.218333 6876 raft_consensus.cc:2454] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:57.218415 7272 raft_consensus.cc:2454] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:57.219123 9337 leader_election.cc:302] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:57.219696 15181 raft_consensus.cc:2781] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.219770 15181 raft_consensus.cc:482] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.219801 15181 raft_consensus.cc:3037] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.220793 15181 raft_consensus.cc:504] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.221105 15181 leader_election.cc:288] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:57.222026 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e10114b061d453c88cee34494da19d5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:57.222172 6877 raft_consensus.cc:3037] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.222476 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e10114b061d453c88cee34494da19d5" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:57.222579 7272 raft_consensus.cc:3037] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.223269 6877 raft_consensus.cc:2454] T 3e10114b061d453c88cee34494da19d5 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:57.223573 7272 raft_consensus.cc:2454] T 3e10114b061d453c88cee34494da19d5 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:58:57.223577 9338 leader_election.cc:302] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:57.223718 15181 raft_consensus.cc:2781] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.223820 15181 raft_consensus.cc:686] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:57.223940 15181 consensus_queue.cc:237] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.225953 6733 catalog_manager.cc:5484] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:57.246893 9266 consensus_queue.cc:237] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:57.248265 9535 raft_consensus.cc:1261] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:57.248294 7138 raft_consensus.cc:1261] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:57.248687 15611 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:57.249298 15611 consensus_queue.cc:1035] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:57.250906 15670 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:57.251438 9534 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:57.252480 7138 raft_consensus.cc:2932] T ae59405a605c4b30a508002a9df30945 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:57.254724 6733 catalog_manager.cc:5484] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:57.258344 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet ae59405a605c4b30a508002a9df30945 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:57.258577 15716 tablet_replica.cc:330] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:57.258724 15716 raft_consensus.cc:2229] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.258850 15716 raft_consensus.cc:2258] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.259321 15716 ts_tablet_manager.cc:1875] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:57.261512 15716 ts_tablet_manager.cc:1888] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:57.261883 15716 log.cc:1196] T ae59405a605c4b30a508002a9df30945 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ae59405a605c4b30a508002a9df30945
I20240501 07:58:57.262558 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet ae59405a605c4b30a508002a9df30945 (table rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]) successfully deleted
I20240501 07:58:57.316372 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:57.325778 9403 raft_consensus.cc:1261] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.326229 7138 raft_consensus.cc:1261] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.327240 15206 consensus_queue.cc:1035] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.328878 14505 consensus_queue.cc:1035] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.334331 6732 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]). Expected version 1 got 0
I20240501 07:58:57.336002 6731 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]). Expected version 1 got 0
I20240501 07:58:57.336335 15932 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.337028 15932 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:57.338660 15692 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.338953 15692 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:57.346127 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a] alter complete (version 1)
I20240501 07:58:57.359488 15933 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.359963 15933 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:57.361131 6731 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a]). Expected version 1 got 0
I20240501 07:58:57.414003 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.416070 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:57.417393 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.418573 9382 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:57.419258 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.423148 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:57.439198 7121 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.451079 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:57.457634 15932 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:57.457978 15932 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:57.458068 15933 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:57.458323 15933 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:57.462055 15692 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:57.462343 15692 tablet.cc:1620] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:57.464743 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=a542026d805d4bf29221e863a6a9292a] alter complete (version 2)
I20240501 07:58:57.475445 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:57.516844 9403 consensus_queue.cc:237] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.518378 9665 raft_consensus.cc:1261] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:57.518409 7272 raft_consensus.cc:1261] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:57.518779 12912 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:57.518951 15181 consensus_queue.cc:1035] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:57.521708 15181 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:57.522244 9665 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:57.523448 7272 raft_consensus.cc:2932] T c72ca77c85c94540905f236a4fc2e300 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:57.525609 6728 catalog_manager.cc:5484] T c72ca77c85c94540905f236a4fc2e300 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:57.529220 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet c72ca77c85c94540905f236a4fc2e300 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:57.529285 15021 consensus_queue.cc:1035] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.529462 15835 tablet_replica.cc:330] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:57.529618 15835 raft_consensus.cc:2229] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.529758 15835 raft_consensus.cc:2258] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.530117 15835 ts_tablet_manager.cc:1875] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:57.532219 15835 ts_tablet_manager.cc:1888] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:57.532272 15835 log.cc:1196] T c72ca77c85c94540905f236a4fc2e300 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c72ca77c85c94540905f236a4fc2e300
I20240501 07:58:57.532903 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet c72ca77c85c94540905f236a4fc2e300 (table rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]) successfully deleted
I20240501 07:58:57.534775 15684 consensus_queue.cc:1035] T d0ed3a7796e548e781898ef10c2601bb P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.566082 15181 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.566184 15181 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:57.566573 15181 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:57.567021 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:57.567152 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:57.567302 7272 raft_consensus.cc:2379] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa6964452854415a85ad72f8639768d6 in current term 1: Already voted for candidate ac6c8bc1b74b483791e262abee7862cf in this term.
I20240501 07:58:57.567754 9339 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:57.568032 15181 raft_consensus.cc:3037] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.569044 15181 raft_consensus.cc:2726] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:57.569608 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:57.575198 15932 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:57.575629 15692 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:57.576406 15933 tablet.cc:1722] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:57.580803 15445 consensus_queue.cc:1035] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.584463 14116 consensus_queue.cc:1035] T f560797734bc458ea6d9c538d8bacf7d P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.585103 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=a542026d805d4bf29221e863a6a9292a] alter complete (version 3)
I20240501 07:58:57.619684 15965 consensus_queue.cc:1035] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.623883 12912 consensus_queue.cc:1035] T 3e10114b061d453c88cee34494da19d5 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.658080 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.658851 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.668676 6857 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:58:57.672163 7121 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.673409 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:57.673638 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:57.675423 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:57.683461 9382 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:58:57.687743 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:57.687896 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:57.693039 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 001a2aed4ca94c81a4f350da98ae8132
I20240501 07:58:57.693774 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:57.693969 15716 tablet_replica.cc:330] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:57.694126 15716 raft_consensus.cc:2229] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.694240 15716 raft_consensus.cc:2258] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.695031 15716 ts_tablet_manager.cc:1875] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.697216 15716 ts_tablet_manager.cc:1888] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.697275 15716 log.cc:1196] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/001a2aed4ca94c81a4f350da98ae8132
I20240501 07:58:57.697522 15716 ts_tablet_manager.cc:1909] T 001a2aed4ca94c81a4f350da98ae8132 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:57.698184 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:57.698506 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table_new_name [id=a542026d805d4bf29221e863a6a9292a]) successfully deleted
I20240501 07:58:57.698535 16028 tablet_replica.cc:330] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:57.698662 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 001a2aed4ca94c81a4f350da98ae8132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:57 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:57.698680 16028 raft_consensus.cc:2229] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:57.698853 16028 raft_consensus.cc:2258] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.698956 16029 tablet_replica.cc:330] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:57.699151 16029 raft_consensus.cc:2229] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.699282 16029 raft_consensus.cc:2258] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.699836 16029 ts_tablet_manager.cc:1875] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.701841 16029 ts_tablet_manager.cc:1888] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.701936 16029 log.cc:1196] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/001a2aed4ca94c81a4f350da98ae8132
I20240501 07:58:57.702255 16029 ts_tablet_manager.cc:1909] T 001a2aed4ca94c81a4f350da98ae8132 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:57.703526 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table_new_name [id=a542026d805d4bf29221e863a6a9292a]) successfully deleted
I20240501 07:58:57.708940 16028 ts_tablet_manager.cc:1875] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:57.711468 16028 ts_tablet_manager.cc:1888] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:57.711588 16028 log.cc:1196] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/001a2aed4ca94c81a4f350da98ae8132
I20240501 07:58:57.711882 16028 ts_tablet_manager.cc:1909] T 001a2aed4ca94c81a4f350da98ae8132 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:57.712495 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 001a2aed4ca94c81a4f350da98ae8132 (table rebalancer_dynamic_table_new_name [id=a542026d805d4bf29221e863a6a9292a]) successfully deleted
I20240501 07:58:57.750123 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:57.756201 9382 tablet_service.cc:1467] Processing CreateTablet for tablet c012af1775ea40bdb0ea78a03fb1d10b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.756285 7121 tablet_service.cc:1467] Processing CreateTablet for tablet c012af1775ea40bdb0ea78a03fb1d10b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.756508 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c012af1775ea40bdb0ea78a03fb1d10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.756541 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c012af1775ea40bdb0ea78a03fb1d10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.759070 6859 tablet_service.cc:1467] Processing CreateTablet for tablet 07fdc1e114774fdb8c84adfdbf1d63f2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.759114 6857 tablet_service.cc:1467] Processing CreateTablet for tablet c012af1775ea40bdb0ea78a03fb1d10b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.759122 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 07fdc1e114774fdb8c84adfdbf1d63f2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.759351 6859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fdc1e114774fdb8c84adfdbf1d63f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.759363 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fdc1e114774fdb8c84adfdbf1d63f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.760356 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 6171adbaa9b4481988269d9a57220deb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.760495 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 6171adbaa9b4481988269d9a57220deb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.760617 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6171adbaa9b4481988269d9a57220deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.760717 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6171adbaa9b4481988269d9a57220deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.761536 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c012af1775ea40bdb0ea78a03fb1d10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.761749 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 6171adbaa9b4481988269d9a57220deb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.761945 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6171adbaa9b4481988269d9a57220deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.762814 16043 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:57.763484 16044 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:57.763988 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 07fdc1e114774fdb8c84adfdbf1d63f2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=9e62bf29730743a29b8b3d6e783a0ca2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.764102 16043 tablet_bootstrap.cc:654] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.764238 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fdc1e114774fdb8c84adfdbf1d63f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.764366 16044 tablet_bootstrap.cc:654] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.764744 16040 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:57.765199 16043 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:57.765295 16043 ts_tablet_manager.cc:1367] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.765738 16043 raft_consensus.cc:348] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.765857 16043 raft_consensus.cc:374] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.765906 16043 raft_consensus.cc:729] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.766026 16044 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:57.766113 16044 ts_tablet_manager.cc:1367] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.766140 16041 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:57.766678 16044 raft_consensus.cc:348] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.766844 16044 raft_consensus.cc:374] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.766894 16044 raft_consensus.cc:729] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.767061 16044 consensus_queue.cc:260] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.767333 16040 tablet_bootstrap.cc:654] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.767473 16044 ts_tablet_manager.cc:1398] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.767976 16044 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:57.769374 16044 tablet_bootstrap.cc:654] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.769409 16041 tablet_bootstrap.cc:654] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.769954 16043 consensus_queue.cc:260] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.770417 16043 ts_tablet_manager.cc:1398] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:57.770948 16040 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:57.771088 16040 ts_tablet_manager.cc:1367] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:57.771404 16044 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:57.771529 16044 ts_tablet_manager.cc:1367] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:57.771566 16040 raft_consensus.cc:348] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.771687 16040 raft_consensus.cc:374] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.771739 16040 raft_consensus.cc:729] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.771857 16040 consensus_queue.cc:260] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.771996 16044 raft_consensus.cc:348] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.772148 16044 raft_consensus.cc:374] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.772226 16044 raft_consensus.cc:729] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.772861 16040 ts_tablet_manager.cc:1398] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:57.773557 16041 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:57.773696 16041 ts_tablet_manager.cc:1367] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:57.774165 16041 raft_consensus.cc:348] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.774257 16044 consensus_queue.cc:260] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.774324 16041 raft_consensus.cc:374] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.774412 16041 raft_consensus.cc:729] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.774580 16041 consensus_queue.cc:260] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.775089 16041 ts_tablet_manager.cc:1398] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.775343 16041 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:57.775583 16042 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:57.776535 16042 tablet_bootstrap.cc:654] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.777019 16041 tablet_bootstrap.cc:654] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.778100 16041 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:57.778173 16041 ts_tablet_manager.cc:1367] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.778415 16044 ts_tablet_manager.cc:1398] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:57.778620 16041 raft_consensus.cc:348] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.778708 16041 raft_consensus.cc:374] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.778738 16041 raft_consensus.cc:729] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.778846 16041 consensus_queue.cc:260] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.779160 16041 ts_tablet_manager.cc:1398] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.780287 16042 tablet_bootstrap.cc:492] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:57.780369 16042 ts_tablet_manager.cc:1367] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:57.780812 16042 raft_consensus.cc:348] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.780967 16042 raft_consensus.cc:374] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.781031 16042 raft_consensus.cc:729] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.787382 16042 consensus_queue.cc:260] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.787673 7138 consensus_queue.cc:237] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:57.787760 16042 ts_tablet_manager.cc:1398] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:58:57.787910 16042 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:57.788796 16042 tablet_bootstrap.cc:654] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.789496 9666 raft_consensus.cc:1261] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:57.789580 6876 raft_consensus.cc:1261] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:57.790004 15021 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:57.790318 15021 consensus_queue.cc:1035] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:57.791654 15181 raft_consensus.cc:482] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.791733 15181 raft_consensus.cc:504] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.792016 15181 leader_election.cc:288] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:57.792418 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:57.792527 9535 raft_consensus.cc:2454] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:58:57.792848 9339 leader_election.cc:302] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:58:57.792788 15684 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:57.793161 15181 raft_consensus.cc:2781] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.793205 15181 raft_consensus.cc:482] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.793236 15181 raft_consensus.cc:3037] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.793259 6876 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:57.794216 15181 raft_consensus.cc:504] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.794517 15181 leader_election.cc:288] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:57.795091 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:57.795250 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:57.795339 9535 raft_consensus.cc:3037] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:58:57.795564 9338 leader_election.cc:341] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:57.795820 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:57.796232 9535 raft_consensus.cc:2454] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
W20240501 07:58:57.796300 9338 leader_election.cc:341] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:57.796501 9339 leader_election.cc:302] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters: 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:57.796731 15181 raft_consensus.cc:2781] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.796655 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:57.796804 15181 raft_consensus.cc:686] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:58:57.796918 15181 consensus_queue.cc:237] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.799199 6729 catalog_manager.cc:5484] T c2fa7739592145d5b90d98fecdbe02b2 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:57.799371 9666 raft_consensus.cc:2932] T c2fa7739592145d5b90d98fecdbe02b2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:57.799873 16042 tablet_bootstrap.cc:492] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:57.799957 16042 ts_tablet_manager.cc:1367] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:57.800381 16042 raft_consensus.cc:348] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.800504 16042 raft_consensus.cc:374] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.800544 16042 raft_consensus.cc:729] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.800673 16042 consensus_queue.cc:260] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:57.800699 6736 catalog_manager.cc:5484] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:57.801008 16042 ts_tablet_manager.cc:1398] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.801152 16042 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:57.805577 16042 tablet_bootstrap.cc:654] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.806389 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet c2fa7739592145d5b90d98fecdbe02b2 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:57.808271 16042 tablet_bootstrap.cc:492] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:57.808358 16042 ts_tablet_manager.cc:1367] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 07:58:57.808780 16042 raft_consensus.cc:348] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.808877 16042 raft_consensus.cc:374] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.808907 16042 raft_consensus.cc:729] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.809007 16042 consensus_queue.cc:260] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.809264 16042 ts_tablet_manager.cc:1398] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.810132 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 3097ee547e074933987e03142853413d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]), partition=
I20240501 07:58:57.810395 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3097ee547e074933987e03142853413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.812343 16071 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:57.812517 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 3097ee547e074933987e03142853413d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]), partition=
I20240501 07:58:57.812723 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3097ee547e074933987e03142853413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.813125 16071 tablet_bootstrap.cc:654] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.814841 16073 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:57.814858 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 3097ee547e074933987e03142853413d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]), partition=
I20240501 07:58:57.815068 16071 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:57.815119 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3097ee547e074933987e03142853413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.815141 16071 ts_tablet_manager.cc:1367] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.815560 16071 raft_consensus.cc:348] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.815661 16071 raft_consensus.cc:374] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.815699 16071 raft_consensus.cc:729] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.815757 16073 tablet_bootstrap.cc:654] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.815836 16071 consensus_queue.cc:260] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.816140 16071 ts_tablet_manager.cc:1398] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.816728 16073 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:57.816751 16043 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:57.816802 16073 ts_tablet_manager.cc:1367] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:57.817270 16073 raft_consensus.cc:348] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.817363 16073 raft_consensus.cc:374] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.817392 16073 raft_consensus.cc:729] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.817643 16043 tablet_bootstrap.cc:654] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.818691 16016 raft_consensus.cc:482] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.817523 16073 consensus_queue.cc:260] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.818779 16016 raft_consensus.cc:504] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:57.819168 16016 leader_election.cc:288] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:57.819466 16073 ts_tablet_manager.cc:1398] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:57.820058 16043 tablet_bootstrap.cc:492] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:57.820140 16043 ts_tablet_manager.cc:1367] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.820241 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:57.820240 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171adbaa9b4481988269d9a57220deb" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:57.820410 9535 raft_consensus.cc:2379] T 6171adbaa9b4481988269d9a57220deb P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1a9029bee6a144938f68b363e4cca162 in current term 1: Already voted for candidate fa6964452854415a85ad72f8639768d6 in this term.
I20240501 07:58:57.820600 16043 raft_consensus.cc:348] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.820711 16043 raft_consensus.cc:374] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.820734 16043 raft_consensus.cc:729] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.820729 6815 leader_election.cc:302] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162; no voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6
I20240501 07:58:57.820834 16043 consensus_queue.cc:260] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.821112 16043 ts_tablet_manager.cc:1398] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.821682 15220 raft_consensus.cc:3037] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.822750 15220 raft_consensus.cc:2726] T 6171adbaa9b4481988269d9a57220deb P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:57.827247 16070 tablet_replica.cc:330] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:57.827384 16070 raft_consensus.cc:2229] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:57.827507 16070 raft_consensus.cc:2258] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:57.827894 16070 ts_tablet_manager.cc:1875] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:57.829785 16070 ts_tablet_manager.cc:1888] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:57.829854 16070 log.cc:1196] T c2fa7739592145d5b90d98fecdbe02b2 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c2fa7739592145d5b90d98fecdbe02b2
I20240501 07:58:57.830446 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet c2fa7739592145d5b90d98fecdbe02b2 (table rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]) successfully deleted
I20240501 07:58:57.848793 15670 raft_consensus.cc:482] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.848904 15670 raft_consensus.cc:504] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.849258 15670 leader_election.cc:288] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:57.849542 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3097ee547e074933987e03142853413d" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:57.849568 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3097ee547e074933987e03142853413d" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:57.849664 7272 raft_consensus.cc:2454] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:57.849692 7009 raft_consensus.cc:2454] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:58:57.849896 9205 leader_election.cc:302] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:57.850044 15670 raft_consensus.cc:2781] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.850102 15670 raft_consensus.cc:482] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.850137 15670 raft_consensus.cc:3037] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.850947 15670 raft_consensus.cc:504] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.851312 15670 leader_election.cc:288] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:58:57.851559 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3097ee547e074933987e03142853413d" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:57.851601 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3097ee547e074933987e03142853413d" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:57.851665 7272 raft_consensus.cc:3037] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.851718 7009 raft_consensus.cc:3037] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.852545 7009 raft_consensus.cc:2454] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:57.852547 7272 raft_consensus.cc:2454] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:58:57.852777 9205 leader_election.cc:302] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:57.853225 15670 raft_consensus.cc:2781] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.853307 15670 raft_consensus.cc:686] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:58:57.853426 15670 consensus_queue.cc:237] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.855518 6736 catalog_manager.cc:5484] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:57.924492 15220 raft_consensus.cc:482] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.924604 15220 raft_consensus.cc:504] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.924976 15220 leader_election.cc:288] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:57.925261 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fdc1e114774fdb8c84adfdbf1d63f2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:57.925261 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fdc1e114774fdb8c84adfdbf1d63f2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:57.925398 9535 raft_consensus.cc:2454] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:57.925398 9271 raft_consensus.cc:2454] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:57.925637 6814 leader_election.cc:302] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:57.925812 15220 raft_consensus.cc:2781] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.925870 15220 raft_consensus.cc:482] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.925916 15220 raft_consensus.cc:3037] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.926766 15220 raft_consensus.cc:504] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.927111 15220 leader_election.cc:288] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:57.927395 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fdc1e114774fdb8c84adfdbf1d63f2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:57.927395 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fdc1e114774fdb8c84adfdbf1d63f2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:57.927498 9271 raft_consensus.cc:3037] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.927507 9535 raft_consensus.cc:3037] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.928385 9271 raft_consensus.cc:2454] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:57.928380 9535 raft_consensus.cc:2454] T 07fdc1e114774fdb8c84adfdbf1d63f2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:57.928639 6814 leader_election.cc:302] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:58:57.928783 15220 raft_consensus.cc:2781] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.928869 15220 raft_consensus.cc:686] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:57.928999 15220 consensus_queue.cc:237] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:57.931190 6731 catalog_manager.cc:5484] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:57.962606 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:57.966537 7009 raft_consensus.cc:1261] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.966537 7272 raft_consensus.cc:1261] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.966905 15611 consensus_queue.cc:1035] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.967096 15670 consensus_queue.cc:1035] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.970042 6734 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]). Expected version 1 got 0
I20240501 07:58:57.970598 6736 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]). Expected version 1 got 0
I20240501 07:58:57.970649 16120 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.971256 16120 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:57.972016 16121 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.972324 16121 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:57.973269 6731 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]). Expected version 1 got 0
I20240501 07:58:57.975842 16122 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:57.976213 16122 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:57.979216 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf] alter complete (version 1)
I20240501 07:58:57.979571 6731 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf]). Expected version 1 got 0
I20240501 07:58:58.061779 16062 raft_consensus.cc:482] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.061904 16062 raft_consensus.cc:504] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.062327 16062 leader_election.cc:288] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:58.062739 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c012af1775ea40bdb0ea78a03fb1d10b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:58.062867 6876 raft_consensus.cc:2454] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:58.063107 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c012af1775ea40bdb0ea78a03fb1d10b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:58.063133 7076 leader_election.cc:302] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:58.063256 9403 raft_consensus.cc:2454] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:58.063354 16062 raft_consensus.cc:2781] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.063419 16062 raft_consensus.cc:482] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.063459 16062 raft_consensus.cc:3037] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.064549 16062 raft_consensus.cc:504] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.064910 16062 leader_election.cc:288] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:58.065311 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c012af1775ea40bdb0ea78a03fb1d10b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:58.065359 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c012af1775ea40bdb0ea78a03fb1d10b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:58.065423 6876 raft_consensus.cc:3037] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.065507 9403 raft_consensus.cc:3037] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.066398 6876 raft_consensus.cc:2454] T c012af1775ea40bdb0ea78a03fb1d10b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:58.066533 9403 raft_consensus.cc:2454] T c012af1775ea40bdb0ea78a03fb1d10b P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:58.066666 7076 leader_election.cc:302] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:58.066861 16062 raft_consensus.cc:2781] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.067085 16062 raft_consensus.cc:686] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:58.067427 16062 consensus_queue.cc:237] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.069630 6734 catalog_manager.cc:5484] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.083593 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:58.091327 16120 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.091653 16120 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:58:58.091944 16121 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.092237 16121 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:58:58.094152 16122 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.094452 16122 tablet.cc:1620] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:58.102325 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=e3b522faf74a4a8c99c9ed48e6d0cecf] alter complete (version 2)
I20240501 07:58:58.103086 9238 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:58.104940 6858 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:58.114248 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.122807 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:58.128191 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.132508 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:58.137420 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:58.163462 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.206678 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:58.212082 16120 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.212888 16122 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.214819 16121 tablet.cc:1722] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.219863 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=e3b522faf74a4a8c99c9ed48e6d0cecf] alter complete (version 3)
I20240501 07:58:58.220353 7009 consensus_queue.cc:237] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.221717 7272 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:58.221920 9271 raft_consensus.cc:1261] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:58.222263 15717 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:58.222440 16082 consensus_queue.cc:1035] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:58.224213 15717 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:58.226181 9271 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:58.229164 6729 catalog_manager.cc:5484] T 93af29aad2ff498dbd8bccaba12f6a70 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:58.230794 7272 raft_consensus.cc:2932] T 93af29aad2ff498dbd8bccaba12f6a70 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:58.232676 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 93af29aad2ff498dbd8bccaba12f6a70 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:58.233065 16160 tablet_replica.cc:330] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:58.233287 16160 raft_consensus.cc:2229] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.233497 16160 raft_consensus.cc:2258] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.234071 16160 ts_tablet_manager.cc:1875] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:58.237437 16160 ts_tablet_manager.cc:1888] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:58:58.237532 16160 log.cc:1196] T 93af29aad2ff498dbd8bccaba12f6a70 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/93af29aad2ff498dbd8bccaba12f6a70
I20240501 07:58:58.238082 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 93af29aad2ff498dbd8bccaba12f6a70 (table rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]) successfully deleted
I20240501 07:58:58.306447 15965 consensus_queue.cc:1035] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.310379 15965 consensus_queue.cc:1035] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.323262 6728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:58.323408 6728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:58.325831 6728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 3097ee547e074933987e03142853413d
I20240501 07:58:58.326558 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 3097ee547e074933987e03142853413d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:58.327119 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 3097ee547e074933987e03142853413d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:58.327198 16070 tablet_replica.cc:330] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:58.327355 16070 raft_consensus.cc:2229] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.327481 16070 raft_consensus.cc:2258] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.327651 7252 tablet_service.cc:1514] Processing DeleteTablet for tablet 3097ee547e074933987e03142853413d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:58:58.327893 16202 tablet_replica.cc:330] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:58.327955 16203 tablet_replica.cc:330] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:58:58.328024 16202 raft_consensus.cc:2229] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:58.328164 16203 raft_consensus.cc:2229] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.328364 16203 raft_consensus.cc:2258] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.328380 16202 raft_consensus.cc:2258] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.328406 16070 ts_tablet_manager.cc:1875] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:58.329005 16203 ts_tablet_manager.cc:1875] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:58.329309 16202 ts_tablet_manager.cc:1875] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:58.330801 16070 ts_tablet_manager.cc:1888] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:58.331617 16070 log.cc:1196] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3097ee547e074933987e03142853413d
I20240501 07:58:58.331925 16070 ts_tablet_manager.cc:1909] T 3097ee547e074933987e03142853413d P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:58.332826 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table_new_name [id=e3b522faf74a4a8c99c9ed48e6d0cecf]) successfully deleted
I20240501 07:58:58.334786 16203 ts_tablet_manager.cc:1888] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:58.334865 16203 log.cc:1196] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3097ee547e074933987e03142853413d
I20240501 07:58:58.335180 16203 ts_tablet_manager.cc:1909] T 3097ee547e074933987e03142853413d P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:58:58.335844 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table_new_name [id=e3b522faf74a4a8c99c9ed48e6d0cecf]) successfully deleted
I20240501 07:58:58.344893 16202 ts_tablet_manager.cc:1888] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:58.345041 16202 log.cc:1196] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3097ee547e074933987e03142853413d
I20240501 07:58:58.345372 16202 ts_tablet_manager.cc:1909] T 3097ee547e074933987e03142853413d P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:58:58.347507 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 3097ee547e074933987e03142853413d (table rebalancer_dynamic_table_new_name [id=e3b522faf74a4a8c99c9ed48e6d0cecf]) successfully deleted
I20240501 07:58:58.350260 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.351027 6858 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:58.355275 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:58.359829 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.360162 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.376467 15220 consensus_queue.cc:1035] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.381443 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:58.391105 15220 consensus_queue.cc:1035] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.399596 6990 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.430321 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:58.432626 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:58.438515 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 37b78e00a38a4c6480a711f310cbb094 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]), partition=
I20240501 07:58:58.438774 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b78e00a38a4c6480a711f310cbb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.440963 16218 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:58.441258 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 37b78e00a38a4c6480a711f310cbb094 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]), partition=
I20240501 07:58:58.441362 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 37b78e00a38a4c6480a711f310cbb094 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]), partition=
I20240501 07:58:58.441500 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b78e00a38a4c6480a711f310cbb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.441577 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b78e00a38a4c6480a711f310cbb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.444988 16218 tablet_bootstrap.cc:654] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.446044 16218 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:58.446121 16218 ts_tablet_manager.cc:1367] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:58.446542 16218 raft_consensus.cc:348] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.446662 16218 raft_consensus.cc:374] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.446700 16218 raft_consensus.cc:729] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.446812 16218 consensus_queue.cc:260] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.447077 16218 ts_tablet_manager.cc:1398] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.448743 16219 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:58.450419 16219 tablet_bootstrap.cc:654] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.451098 16220 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:58:58.451886 16220 tablet_bootstrap.cc:654] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.453415 16220 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:58:58.453512 16220 ts_tablet_manager.cc:1367] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.453944 16220 raft_consensus.cc:348] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.454066 16220 raft_consensus.cc:374] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.454095 16220 raft_consensus.cc:729] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.454207 16220 consensus_queue.cc:260] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.454478 16220 ts_tablet_manager.cc:1398] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.455207 16219 tablet_bootstrap.cc:492] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:58.455298 16219 ts_tablet_manager.cc:1367] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:58.455768 16219 raft_consensus.cc:348] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.455870 16219 raft_consensus.cc:374] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.455909 16219 raft_consensus.cc:729] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.456018 16219 consensus_queue.cc:260] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.456274 16219 ts_tablet_manager.cc:1398] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:58.486510 9666 consensus_queue.cc:237] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:58.487758 9403 raft_consensus.cc:1261] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:58.487782 7271 raft_consensus.cc:1261] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:58.488250 15946 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:58.488448 16217 consensus_queue.cc:1035] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:58.490773 16217 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:58.491528 9403 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:58.494570 6731 catalog_manager.cc:5484] T 748b249371a4464f88aeb0c4c8558998 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:58.494872 7271 raft_consensus.cc:2932] T 748b249371a4464f88aeb0c4c8558998 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:58.496359 16062 raft_consensus.cc:482] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.496454 16062 raft_consensus.cc:504] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.496838 16062 leader_election.cc:288] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:58.498514 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b78e00a38a4c6480a711f310cbb094" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:58.498634 9666 raft_consensus.cc:2454] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:58.498874 7076 leader_election.cc:302] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:58.499274 16062 raft_consensus.cc:2781] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.499352 16062 raft_consensus.cc:482] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.499392 16062 raft_consensus.cc:3037] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.500227 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b78e00a38a4c6480a711f310cbb094" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:58:58.500249 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 748b249371a4464f88aeb0c4c8558998 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:58:58.500361 9403 raft_consensus.cc:2454] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:58.500450 16062 raft_consensus.cc:504] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.500582 16202 tablet_replica.cc:330] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:58:58.500725 16202 raft_consensus.cc:2229] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.500749 16062 leader_election.cc:288] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:58.500864 16202 raft_consensus.cc:2258] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.501168 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b78e00a38a4c6480a711f310cbb094" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:58.501168 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b78e00a38a4c6480a711f310cbb094" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:58.501266 9403 raft_consensus.cc:3037] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.501266 9666 raft_consensus.cc:3037] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.501339 16202 ts_tablet_manager.cc:1875] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:58.502125 9666 raft_consensus.cc:2454] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:58.502130 9403 raft_consensus.cc:2454] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:58.502471 7076 leader_election.cc:302] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:58.502664 16062 raft_consensus.cc:2781] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.502755 16062 raft_consensus.cc:686] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:58.502878 16062 consensus_queue.cc:237] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:58.503283 16202 ts_tablet_manager.cc:1888] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:58.503355 16202 log.cc:1196] T 748b249371a4464f88aeb0c4c8558998 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/748b249371a4464f88aeb0c4c8558998
I20240501 07:58:58.504241 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 748b249371a4464f88aeb0c4c8558998 (table rebalancer_extra_table_14 [id=b24fd09cea5543ed803e43c9dacf4e06]) successfully deleted
I20240501 07:58:58.504449 16062 consensus_queue.cc:1035] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.505403 6736 catalog_manager.cc:5484] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:58.510185 16062 consensus_queue.cc:1035] T c012af1775ea40bdb0ea78a03fb1d10b P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:58:58.540901 7188 tablet.cc:2369] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:58.575201 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:58.584084 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 51f40e32ef7f4757b59aef57320d866b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.584432 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f40e32ef7f4757b59aef57320d866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.585095 6990 tablet_service.cc:1467] Processing CreateTablet for tablet ef07bb51bebe4cd5aa66e2b2fc586148 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.585453 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef07bb51bebe4cd5aa66e2b2fc586148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.587469 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 51f40e32ef7f4757b59aef57320d866b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.587756 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f40e32ef7f4757b59aef57320d866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.587765 16288 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:58.585095 6989 tablet_service.cc:1467] Processing CreateTablet for tablet e4440dde610a499fb2e497c4fd638088 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.588348 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4440dde610a499fb2e497c4fd638088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.588671 9646 tablet_service.cc:1467] Processing CreateTablet for tablet ef07bb51bebe4cd5aa66e2b2fc586148 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.588932 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef07bb51bebe4cd5aa66e2b2fc586148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.588955 16288 tablet_bootstrap.cc:654] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.589929 16290 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:58.591401 9238 tablet_service.cc:1467] Processing CreateTablet for tablet ef07bb51bebe4cd5aa66e2b2fc586148 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.591624 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef07bb51bebe4cd5aa66e2b2fc586148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.591759 16290 tablet_bootstrap.cc:654] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.591923 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 51f40e32ef7f4757b59aef57320d866b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.592164 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f40e32ef7f4757b59aef57320d866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.593403 16287 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:58.594004 16290 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:58.594072 16290 ts_tablet_manager.cc:1367] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:58.594362 16287 tablet_bootstrap.cc:654] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.594357 16292 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:58.594485 6857 tablet_service.cc:1467] Processing CreateTablet for tablet e4440dde610a499fb2e497c4fd638088 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.594523 16290 raft_consensus.cc:348] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.594630 16290 raft_consensus.cc:374] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.594662 16290 raft_consensus.cc:729] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.594710 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4440dde610a499fb2e497c4fd638088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.594791 16290 consensus_queue.cc:260] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.595265 16290 ts_tablet_manager.cc:1398] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.595425 16290 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:58.596347 16290 tablet_bootstrap.cc:654] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.596683 16288 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:58.596805 16288 ts_tablet_manager.cc:1367] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:58.597427 16288 raft_consensus.cc:348] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.597554 16288 raft_consensus.cc:374] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.597599 16288 raft_consensus.cc:729] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.597776 16288 consensus_queue.cc:260] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.598085 16290 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:58.598140 16288 ts_tablet_manager.cc:1398] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.598182 16290 ts_tablet_manager.cc:1367] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:58.598333 16288 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
W20240501 07:58:58.598606 7057 tablet.cc:2369] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:58.598609 16290 raft_consensus.cc:348] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.598709 16290 raft_consensus.cc:374] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.598749 16290 raft_consensus.cc:729] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.598873 16290 consensus_queue.cc:260] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.599215 16290 ts_tablet_manager.cc:1398] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.600229 16219 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:58.600503 7121 tablet_service.cc:1467] Processing CreateTablet for tablet e4440dde610a499fb2e497c4fd638088 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6d593408d8894780b5fbe7cbb688a376]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.600792 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4440dde610a499fb2e497c4fd638088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.601176 16219 tablet_bootstrap.cc:654] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.604074 16288 tablet_bootstrap.cc:654] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.604776 16218 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:58.605616 16218 tablet_bootstrap.cc:654] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.605695 16219 tablet_bootstrap.cc:492] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:58.605783 16219 ts_tablet_manager.cc:1367] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:58.606261 16219 raft_consensus.cc:348] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.606371 16219 raft_consensus.cc:374] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.606405 16219 raft_consensus.cc:729] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.606521 16219 consensus_queue.cc:260] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.606808 16219 ts_tablet_manager.cc:1398] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.606935 16218 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:58.607044 16218 ts_tablet_manager.cc:1367] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.607489 16218 raft_consensus.cc:348] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.607579 16218 raft_consensus.cc:374] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.607609 16218 raft_consensus.cc:729] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.607723 16218 consensus_queue.cc:260] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.608004 16218 ts_tablet_manager.cc:1398] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.614624 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:58.617357 16292 tablet_bootstrap.cc:654] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.619786 9665 raft_consensus.cc:1261] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.619796 9403 raft_consensus.cc:1261] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.619946 16292 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:58.620046 16292 ts_tablet_manager.cc:1367] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:58:58.620612 16292 raft_consensus.cc:348] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.620811 16292 raft_consensus.cc:374] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.620896 16292 raft_consensus.cc:729] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.621249 16292 consensus_queue.cc:260] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.621393 16062 consensus_queue.cc:1035] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.621830 16292 ts_tablet_manager.cc:1398] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:58.622001 16292 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:58.622277 16059 consensus_queue.cc:1035] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.623044 16292 tablet_bootstrap.cc:654] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.624542 16287 tablet_bootstrap.cc:492] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:58.624647 16287 ts_tablet_manager.cc:1367] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.002s
I20240501 07:58:58.625118 16288 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:58.625126 16287 raft_consensus.cc:348] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.625219 16288 ts_tablet_manager.cc:1367] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 07:58:58.625234 16287 raft_consensus.cc:374] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.625265 16287 raft_consensus.cc:729] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.625384 16287 consensus_queue.cc:260] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.625610 6735 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]). Expected version 1 got 0
I20240501 07:58:58.625700 16287 ts_tablet_manager.cc:1398] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:58.626030 16288 raft_consensus.cc:348] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.626159 16288 raft_consensus.cc:374] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.626210 16288 raft_consensus.cc:729] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.626389 16288 consensus_queue.cc:260] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.626818 16288 ts_tablet_manager.cc:1398] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:58.629524 16292 tablet_bootstrap.cc:492] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:58.630429 16292 ts_tablet_manager.cc:1367] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:58.631883 16292 raft_consensus.cc:348] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.632177 16292 raft_consensus.cc:374] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.632270 16292 raft_consensus.cc:729] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.632463 16292 consensus_queue.cc:260] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.632846 16292 ts_tablet_manager.cc:1398] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:58.634397 6735 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]). Expected version 1 got 0
I20240501 07:58:58.637480 6729 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]). Expected version 1 got 0
I20240501 07:58:58.644421 16319 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:58.644860 16319 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:58.647960 6732 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d]). Expected version 1 got 0
I20240501 07:58:58.652400 16321 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:58.652779 16321 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:58.654299 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d] alter complete (version 1)
I20240501 07:58:58.654871 16320 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:58.655320 16320 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
W20240501 07:58:58.685947 6926 tablet.cc:2369] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:58.690297 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.691510 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.692260 9515 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:58:58.695362 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:58.702240 6989 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.703059 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:58.704704 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:58.736995 15220 raft_consensus.cc:482] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.737124 15220 raft_consensus.cc:504] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.737502 15220 leader_election.cc:288] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:58.737849 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4440dde610a499fb2e497c4fd638088" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:58.737980 7138 raft_consensus.cc:2454] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:58.738262 6813 leader_election.cc:302] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:58.738435 15220 raft_consensus.cc:2781] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.738482 15220 raft_consensus.cc:482] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.738533 15220 raft_consensus.cc:3037] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.738926 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4440dde610a499fb2e497c4fd638088" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:58.739101 7009 raft_consensus.cc:2454] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:58.739710 15220 raft_consensus.cc:504] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.740295 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4440dde610a499fb2e497c4fd638088" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:58.740404 7009 raft_consensus.cc:3037] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.740418 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4440dde610a499fb2e497c4fd638088" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:58.740518 7138 raft_consensus.cc:3037] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.741375 7009 raft_consensus.cc:2454] T e4440dde610a499fb2e497c4fd638088 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:58.741454 7138 raft_consensus.cc:2454] T e4440dde610a499fb2e497c4fd638088 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:58.741657 6815 leader_election.cc:302] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:58.741822 16016 raft_consensus.cc:2781] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.741966 16016 raft_consensus.cc:686] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:58.742091 16016 consensus_queue.cc:237] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:58.742902 15220 leader_election.cc:288] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:58.744704 6736 catalog_manager.cc:5484] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:58.745296 15445 raft_consensus.cc:482] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.745405 15445 raft_consensus.cc:504] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.748270 7121 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:58.751308 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:58.751442 9266 raft_consensus.cc:2454] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:58.750998 15445 leader_election.cc:288] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:58.751775 9470 leader_election.cc:302] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:58.751960 15445 raft_consensus.cc:2781] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.752005 15445 raft_consensus.cc:482] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.752048 15445 raft_consensus.cc:3037] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.754109 15445 raft_consensus.cc:504] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.755065 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:58.755174 9266 raft_consensus.cc:3037] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.755160 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:58:58.755275 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:58.755290 6877 raft_consensus.cc:2454] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:58.754492 15445 leader_election.cc:288] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:58:58.756582 9266 raft_consensus.cc:2454] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:58:58.756899 9470 leader_election.cc:302] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters: 1a9029bee6a144938f68b363e4cca162
I20240501 07:58:58.758059 15445 raft_consensus.cc:2781] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.758267 15445 raft_consensus.cc:686] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:58.758433 15445 consensus_queue.cc:237] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.760118 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:58.760900 6735 catalog_manager.cc:5484] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.766760 16319 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.767104 16319 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:58.767797 16320 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.768064 16320 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:58:58.772814 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=ad4800399b3f41e9a6037dfd3af47c0d] alter complete (version 2)
I20240501 07:58:58.773993 16321 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:58.774317 16321 tablet.cc:1620] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:58:58.807572 9665 consensus_queue.cc:237] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:58:58.808842 6877 raft_consensus.cc:1261] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:58.808842 9266 raft_consensus.cc:1261] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:58:58.809253 16217 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:58.809458 15946 consensus_queue.cc:1035] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:58:58.811211 15946 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.811793 9266 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.813390 6877 raft_consensus.cc:2932] T 64e235bbc64a4dca8028ebf89c2b7b5e P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.815210 6736 catalog_manager.cc:5484] T 64e235bbc64a4dca8028ebf89c2b7b5e P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:58:58.819061 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e235bbc64a4dca8028ebf89c2b7b5e with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:58.819710 16359 tablet_replica.cc:330] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:58.819981 16359 raft_consensus.cc:2229] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.820251 16359 raft_consensus.cc:2258] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.820740 16359 ts_tablet_manager.cc:1875] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:58.823302 16359 ts_tablet_manager.cc:1888] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:58:58.823415 16359 log.cc:1196] T 64e235bbc64a4dca8028ebf89c2b7b5e P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/64e235bbc64a4dca8028ebf89c2b7b5e
I20240501 07:58:58.823992 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 64e235bbc64a4dca8028ebf89c2b7b5e (table rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]) successfully deleted
I20240501 07:58:58.874625 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:58.884356 16319 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.885375 16321 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.887760 16320 tablet.cc:1722] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:58.894881 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=ad4800399b3f41e9a6037dfd3af47c0d] alter complete (version 3)
I20240501 07:58:58.975915 16082 raft_consensus.cc:482] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.976047 16082 raft_consensus.cc:504] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.976626 16082 leader_election.cc:288] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:58.976979 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef07bb51bebe4cd5aa66e2b2fc586148" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:58.976997 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef07bb51bebe4cd5aa66e2b2fc586148" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:58.977118 9266 raft_consensus.cc:2454] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:58.977118 9665 raft_consensus.cc:2454] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:58.977389 6945 leader_election.cc:302] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:58.977727 16082 raft_consensus.cc:2781] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.977792 16082 raft_consensus.cc:482] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.977821 16082 raft_consensus.cc:3037] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.978982 16082 raft_consensus.cc:504] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.979367 16082 leader_election.cc:288] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:58.979838 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef07bb51bebe4cd5aa66e2b2fc586148" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:58.979838 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef07bb51bebe4cd5aa66e2b2fc586148" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:58.979959 9665 raft_consensus.cc:3037] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.979959 9266 raft_consensus.cc:3037] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.980917 9266 raft_consensus.cc:2454] T ef07bb51bebe4cd5aa66e2b2fc586148 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:58.980938 9665 raft_consensus.cc:2454] T ef07bb51bebe4cd5aa66e2b2fc586148 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:58.981251 6945 leader_election.cc:302] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:58.981406 16082 raft_consensus.cc:2781] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.981587 16082 raft_consensus.cc:686] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:58.981746 16082 consensus_queue.cc:237] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:58.984016 6734 catalog_manager.cc:5484] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:58.999333 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:58.999496 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:59.003890 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 37b78e00a38a4c6480a711f310cbb094
I20240501 07:58:59.005149 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b78e00a38a4c6480a711f310cbb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:59.005177 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b78e00a38a4c6480a711f310cbb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:58:59.005188 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b78e00a38a4c6480a711f310cbb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:58 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:58:59.005502 16411 tablet_replica.cc:330] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:58:59.005574 16410 tablet_replica.cc:330] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:59.005759 16411 raft_consensus.cc:2229] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:59.005805 16410 raft_consensus.cc:2229] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:59.005965 16411 raft_consensus.cc:2258] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.006829 16411 ts_tablet_manager.cc:1875] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.006389 16410 raft_consensus.cc:2258] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.010010 16411 ts_tablet_manager.cc:1888] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.010228 16410 ts_tablet_manager.cc:1875] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.010135 16411 log.cc:1196] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/37b78e00a38a4c6480a711f310cbb094
I20240501 07:58:59.010849 16411 ts_tablet_manager.cc:1909] T 37b78e00a38a4c6480a711f310cbb094 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:58:59.011646 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table_new_name [id=ad4800399b3f41e9a6037dfd3af47c0d]) successfully deleted
I20240501 07:58:59.012894 16410 ts_tablet_manager.cc:1888] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.013036 16410 log.cc:1196] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/37b78e00a38a4c6480a711f310cbb094
I20240501 07:58:59.013423 16410 ts_tablet_manager.cc:1909] T 37b78e00a38a4c6480a711f310cbb094 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:59.014142 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table_new_name [id=ad4800399b3f41e9a6037dfd3af47c0d]) successfully deleted
I20240501 07:58:59.016314 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.022015 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.024658 9515 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:58:59.024771 16412 tablet_replica.cc:330] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:58:59.024994 16412 raft_consensus.cc:2229] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:59.025171 16412 raft_consensus.cc:2258] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.027904 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:59.028694 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:59.033679 16412 ts_tablet_manager.cc:1875] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.036458 16412 ts_tablet_manager.cc:1888] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.036618 16412 log.cc:1196] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/37b78e00a38a4c6480a711f310cbb094
I20240501 07:58:59.037019 16412 ts_tablet_manager.cc:1909] T 37b78e00a38a4c6480a711f310cbb094 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:58:59.037904 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 37b78e00a38a4c6480a711f310cbb094 (table rebalancer_dynamic_table_new_name [id=ad4800399b3f41e9a6037dfd3af47c0d]) successfully deleted
I20240501 07:58:59.044574 6989 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:59.045468 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.067478 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.106387 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:59.112985 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 84abf83338e84bbf92a1b41741003845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]), partition=
I20240501 07:58:59.113131 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 84abf83338e84bbf92a1b41741003845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]), partition=
I20240501 07:58:59.113314 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84abf83338e84bbf92a1b41741003845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.113384 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84abf83338e84bbf92a1b41741003845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.115545 16287 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:59.115630 16420 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:59.116158 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 84abf83338e84bbf92a1b41741003845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]), partition=
I20240501 07:58:59.116398 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84abf83338e84bbf92a1b41741003845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.116540 16287 tablet_bootstrap.cc:654] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.116927 16420 tablet_bootstrap.cc:654] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.118036 16287 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:59.118149 16287 ts_tablet_manager.cc:1367] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:59.118441 16288 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:59.118671 16287 raft_consensus.cc:348] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.118783 16287 raft_consensus.cc:374] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.118819 16287 raft_consensus.cc:729] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.118938 16287 consensus_queue.cc:260] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.118956 16420 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:59.119119 16420 ts_tablet_manager.cc:1367] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:59.119284 16288 tablet_bootstrap.cc:654] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.119560 16420 raft_consensus.cc:348] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.119657 16420 raft_consensus.cc:374] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.119684 16420 raft_consensus.cc:729] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.119781 16420 consensus_queue.cc:260] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.120048 16420 ts_tablet_manager.cc:1398] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.120959 16287 ts_tablet_manager.cc:1398] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:59.121848 16288 tablet_bootstrap.cc:492] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:59.121924 16288 ts_tablet_manager.cc:1367] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:59.122364 16288 raft_consensus.cc:348] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.122471 16288 raft_consensus.cc:374] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.122509 16288 raft_consensus.cc:729] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.122632 16288 consensus_queue.cc:260] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.122936 16288 ts_tablet_manager.cc:1398] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.123474 7138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603"
dest_uuid: "fc946354b5d04503a56122e59fc320be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49608
I20240501 07:58:59.123620 7138 raft_consensus.cc:595] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:59.137480 16059 raft_consensus.cc:979] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: : Instructing follower 9a3888360d4d4ace96b6b909c160bfc2 to start an election
I20240501 07:58:59.137617 16059 raft_consensus.cc:1067] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Signalling peer 9a3888360d4d4ace96b6b909c160bfc2 to start an election
I20240501 07:58:59.138187 7009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603"
dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
from {username='slave'} at 127.0.99.131:55099
I20240501 07:58:59.138347 7009 raft_consensus.cc:482] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:59.138406 7009 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.139468 7009 raft_consensus.cc:504] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:59.139937 7009 leader_election.cc:288] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:58:59.139920 6876 raft_consensus.cc:3037] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.140431 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:58:59.140569 9403 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.141427 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:59.141484 9403 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:58:59.141531 7138 raft_consensus.cc:3032] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:59.141566 7138 raft_consensus.cc:729] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:59.141701 7138 consensus_queue.cc:260] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:59.141863 7138 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.142771 7138 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:58:59.143260 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9379f9bf86403ebf1695263db9a603" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:59.143355 6944 leader_election.cc:302] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:59.143383 7271 raft_consensus.cc:3037] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.143522 16082 raft_consensus.cc:2781] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:59.143590 16082 raft_consensus.cc:686] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:59.143715 16082 consensus_queue.cc:237] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:58:59.144356 7271 raft_consensus.cc:2454] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:58:59.147323 14116 consensus_queue.cc:1035] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.148312 6736 catalog_manager.cc:5484] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from fc946354b5d04503a56122e59fc320be (127.0.99.131) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:58:59.151880 15445 consensus_queue.cc:1035] T 51f40e32ef7f4757b59aef57320d866b P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.248090 16333 consensus_queue.cc:1035] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.251462 16468 consensus_queue.cc:1035] T e4440dde610a499fb2e497c4fd638088 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.267264 16397 raft_consensus.cc:482] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.267408 16397 raft_consensus.cc:504] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.267874 16397 leader_election.cc:288] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:59.268165 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84abf83338e84bbf92a1b41741003845" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:59.268225 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84abf83338e84bbf92a1b41741003845" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:59.268319 9535 raft_consensus.cc:2454] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:59.268378 7138 raft_consensus.cc:2454] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:58:59.268675 6946 leader_election.cc:302] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:59.268870 16397 raft_consensus.cc:2781] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.268925 16397 raft_consensus.cc:482] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.268951 16397 raft_consensus.cc:3037] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.269941 16397 raft_consensus.cc:504] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.270242 16397 leader_election.cc:288] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:59.270572 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84abf83338e84bbf92a1b41741003845" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:59.270691 9535 raft_consensus.cc:3037] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.270665 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84abf83338e84bbf92a1b41741003845" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:59.270792 7138 raft_consensus.cc:3037] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.271562 9535 raft_consensus.cc:2454] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:59.271624 7138 raft_consensus.cc:2454] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:58:59.271862 6946 leader_election.cc:302] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:59.272023 16397 raft_consensus.cc:2781] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.272109 16397 raft_consensus.cc:686] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:58:59.272235 16397 consensus_queue.cc:237] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.275040 6729 catalog_manager.cc:5484] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:59.296861 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.298454 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.305044 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.305297 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:59.311642 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.324832 7121 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.326922 9515 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:58:59.345196 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:59.416308 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:58:59.421123 7138 raft_consensus.cc:1261] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.421103 9535 raft_consensus.cc:1261] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.421569 16082 consensus_queue.cc:1035] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.422142 16082 consensus_queue.cc:1035] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.424583 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]). Expected version 1 got 0
I20240501 07:58:59.425343 16510 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:59.425488 6732 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]). Expected version 1 got 0
I20240501 07:58:59.425814 16510 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:59.427949 16432 mvcc.cc:205] Tried to move back new op lower bound from 7022798190263046144 to 7022798189662236672. Current Snapshot: MvccSnapshot[applied={T|T < 7022798190263046144}]
I20240501 07:58:59.430409 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]). Expected version 1 got 0
I20240501 07:58:59.430440 16512 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:59.430930 16512 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:59.431114 6732 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46]). Expected version 1 got 0
I20240501 07:58:59.431975 16513 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:58:59.432310 16513 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:59.433802 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46] alter complete (version 1)
I20240501 07:58:59.465622 16397 consensus_queue.cc:1035] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.469743 16082 consensus_queue.cc:1035] T ef07bb51bebe4cd5aa66e2b2fc586148 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.510393 9515 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:58:59.519924 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.523311 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.524577 7252 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.528975 7121 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.538882 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:58:59.539446 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.543408 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:59.551721 16510 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:59.552042 16510 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:58:59.555258 16513 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:59.555536 16513 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:58:59.556828 16512 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:58:59.557159 16512 tablet.cc:1620] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:58:59.560750 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d2cb8addb97b421fa9b5a240d24daf46] alter complete (version 2)
I20240501 07:58:59.568100 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:59.577517 9403 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:59.578342 16397 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:59.580987 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:58:59.581962 9515 tablet_service.cc:1467] Processing CreateTablet for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.582299 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.583755 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c374464c8fd741fc92645d35e191a644 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.583755 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5805339d2c48699ece42341cd34025 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.584085 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c374464c8fd741fc92645d35e191a644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.584228 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5805339d2c48699ece42341cd34025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.584977 7251 tablet_service.cc:1467] Processing CreateTablet for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.585278 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.586319 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5805339d2c48699ece42341cd34025 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.586547 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5805339d2c48699ece42341cd34025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.587158 16287 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:58:59.588227 16287 tablet_bootstrap.cc:654] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.588802 7250 tablet_service.cc:1467] Processing CreateTablet for tablet c374464c8fd741fc92645d35e191a644 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.589005 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c374464c8fd741fc92645d35e191a644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.589082 9238 tablet_service.cc:1467] Processing CreateTablet for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.589367 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4908fb61c5d4ffdbdc9a22cc3457dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.590896 16420 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:59.591845 16420 tablet_bootstrap.cc:654] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.592371 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5805339d2c48699ece42341cd34025 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.592639 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5805339d2c48699ece42341cd34025. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.592856 6989 tablet_service.cc:1467] Processing CreateTablet for tablet c374464c8fd741fc92645d35e191a644 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=000fa0c96a8c4b19b2bf0de9ce06243a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.593111 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c374464c8fd741fc92645d35e191a644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.598855 16288 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:59.600036 16288 tablet_bootstrap.cc:654] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.600062 16551 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:59.600971 16549 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:59.601816 16549 tablet_bootstrap.cc:654] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.605741 16420 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:59.605839 16420 ts_tablet_manager.cc:1367] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:58:59.606281 16420 raft_consensus.cc:348] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.606384 16420 raft_consensus.cc:374] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.606410 16420 raft_consensus.cc:729] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.606519 16420 consensus_queue.cc:260] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.606775 16420 ts_tablet_manager.cc:1398] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.606969 16420 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:58:59.607898 16420 tablet_bootstrap.cc:654] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.611084 16420 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:58:59.611172 16420 ts_tablet_manager.cc:1367] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:59.612973 16420 raft_consensus.cc:348] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.613396 16420 raft_consensus.cc:374] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.613473 16420 raft_consensus.cc:729] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.613629 16420 consensus_queue.cc:260] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.614746 16420 ts_tablet_manager.cc:1398] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:58:59.615315 16551 tablet_bootstrap.cc:654] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.617707 16551 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:59.617795 16551 ts_tablet_manager.cc:1367] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:58:59.618229 16551 raft_consensus.cc:348] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.618333 16551 raft_consensus.cc:374] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.618361 16551 raft_consensus.cc:729] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.618474 16551 consensus_queue.cc:260] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.618717 16551 ts_tablet_manager.cc:1398] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.623217 16553 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:58:59.623827 16287 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:58:59.623955 16287 ts_tablet_manager.cc:1367] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240501 07:58:59.624099 16288 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:59.624214 16288 ts_tablet_manager.cc:1367] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 07:58:59.624944 16288 raft_consensus.cc:348] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.624967 16287 raft_consensus.cc:348] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.625062 16288 raft_consensus.cc:374] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.625062 16287 raft_consensus.cc:374] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.625095 16288 raft_consensus.cc:729] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.625095 16287 raft_consensus.cc:729] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.625216 16288 consensus_queue.cc:260] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.625216 16287 consensus_queue.cc:260] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.625494 16287 ts_tablet_manager.cc:1398] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:59.625515 16288 ts_tablet_manager.cc:1398] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:59.631865 16549 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:59.631976 16549 ts_tablet_manager.cc:1367] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240501 07:58:59.632505 16549 raft_consensus.cc:348] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.632629 16549 raft_consensus.cc:374] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.632665 16549 raft_consensus.cc:729] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.632786 16549 consensus_queue.cc:260] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.633121 16549 ts_tablet_manager.cc:1398] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:59.633278 16549 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:59.635744 16549 tablet_bootstrap.cc:654] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.637488 14116 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.637589 14116 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.638115 14116 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:59.638373 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:59.638504 9271 raft_consensus.cc:2454] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:58:59.638617 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:59.638808 9470 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters:
W20240501 07:58:59.639089 9469 leader_election.cc:341] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:59.639693 14116 raft_consensus.cc:2781] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.639751 14116 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.639791 14116 raft_consensus.cc:3037] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.640616 14116 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.640916 14116 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:58:59.641376 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:59.641377 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:59.641484 9271 raft_consensus.cc:3037] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.642434 9271 raft_consensus.cc:2454] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
W20240501 07:58:59.642915 9469 leader_election.cc:341] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:58:59.643231 9470 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf; no voters: 15ab96f6477b4291932409b80af07213
I20240501 07:58:59.644057 14116 raft_consensus.cc:2781] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.644165 14116 raft_consensus.cc:686] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:58:59.644305 14116 consensus_queue.cc:237] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.646600 6730 catalog_manager.cc:5484] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:59.646886 16553 tablet_bootstrap.cc:654] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.648613 16553 tablet_bootstrap.cc:492] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:58:59.648710 16553 ts_tablet_manager.cc:1367] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:58:59.649168 16553 raft_consensus.cc:348] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.649269 16553 raft_consensus.cc:374] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.649304 16553 raft_consensus.cc:729] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.649461 16553 consensus_queue.cc:260] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.650961 16553 ts_tablet_manager.cc:1398] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:59.651733 16549 tablet_bootstrap.cc:492] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:59.651829 16549 ts_tablet_manager.cc:1367] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:58:59.652370 16549 raft_consensus.cc:348] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.652484 16549 raft_consensus.cc:374] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.652521 16549 raft_consensus.cc:729] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.652648 16549 consensus_queue.cc:260] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.652722 6876 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa"
dest_uuid: "1a9029bee6a144938f68b363e4cca162"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58064
I20240501 07:58:59.652863 6876 raft_consensus.cc:595] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:59.652962 16549 ts_tablet_manager.cc:1398] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.653148 16549 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:58:59.653879 7271 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:59.654187 16549 tablet_bootstrap.cc:654] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.654232 16397 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:59.655282 16549 tablet_bootstrap.cc:492] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:58:59.655416 16549 ts_tablet_manager.cc:1367] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:59.656086 16549 raft_consensus.cc:348] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.656256 16549 raft_consensus.cc:374] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.656338 16549 raft_consensus.cc:729] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.656520 16549 consensus_queue.cc:260] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.656939 16549 ts_tablet_manager.cc:1398] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.658874 7140 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:58:59.659893 16569 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:59.673934 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:58:59.680991 16510 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:59.681794 16512 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:59.685421 16513 tablet.cc:1722] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:58:59.686614 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=d2cb8addb97b421fa9b5a240d24daf46] alter complete (version 3)
I20240501 07:58:59.693042 15220 raft_consensus.cc:979] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: : Instructing follower 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:58:59.693172 16468 raft_consensus.cc:1067] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Signalling peer 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:58:59.693724 7272 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa"
dest_uuid: "15ab96f6477b4291932409b80af07213"
from {username='slave'} at 127.0.99.129:51117
I20240501 07:58:59.693850 7272 raft_consensus.cc:482] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:59.693912 7272 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.694844 7272 raft_consensus.cc:504] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:59.696900 7272 leader_election.cc:288] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:58:59.697255 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:59.697386 7140 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.697422 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:58:59.697549 9535 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.697664 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0f4ea5ccba462d9468e3f445569eaa" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:58:59.697790 6876 raft_consensus.cc:3032] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:59.697835 6876 raft_consensus.cc:729] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:59.697996 6876 consensus_queue.cc:260] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:59.698231 6876 raft_consensus.cc:3037] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.698403 7140 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:59.698558 9535 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:59.699234 6876 raft_consensus.cc:2454] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:58:59.703730 7208 leader_election.cc:302] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:58:59.703955 16414 raft_consensus.cc:2781] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:59.704146 16414 raft_consensus.cc:686] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:59.704301 16414 consensus_queue.cc:237] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:58:59.706763 6735 catalog_manager.cc:5484] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
W20240501 07:58:59.712199 7319 tablet.cc:2369] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:59.791550 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:59.791705 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:58:59.793773 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 84abf83338e84bbf92a1b41741003845
I20240501 07:58:59.793852 16568 raft_consensus.cc:482] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.793982 16568 raft_consensus.cc:504] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.794409 16568 leader_election.cc:288] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:59.794572 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 84abf83338e84bbf92a1b41741003845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:59 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:58:59.794588 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 84abf83338e84bbf92a1b41741003845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:59 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:58:59.794925 16613 tablet_replica.cc:330] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:58:59.795151 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 84abf83338e84bbf92a1b41741003845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:58:59 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:58:59.795177 16613 raft_consensus.cc:2229] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:59.795209 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5805339d2c48699ece42341cd34025" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:58:59.795315 16613 raft_consensus.cc:2258] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.795351 7272 raft_consensus.cc:2454] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:59.795552 16612 tablet_replica.cc:330] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:58:59.795698 16612 raft_consensus.cc:2229] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:58:59.795799 7075 leader_election.cc:302] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:59.795915 16612 raft_consensus.cc:2258] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.796422 16613 ts_tablet_manager.cc:1875] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.796916 16614 tablet_replica.cc:330] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:58:59.797075 16614 raft_consensus.cc:2229] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:59.797289 16614 raft_consensus.cc:2258] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:59.797618 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5805339d2c48699ece42341cd34025" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:58:59.797773 9666 raft_consensus.cc:2454] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:58:59.798205 16614 ts_tablet_manager.cc:1875] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.798823 16613 ts_tablet_manager.cc:1888] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.798924 16613 log.cc:1196] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/84abf83338e84bbf92a1b41741003845
I20240501 07:58:59.799265 16613 ts_tablet_manager.cc:1909] T 84abf83338e84bbf92a1b41741003845 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:58:59.800359 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table_new_name [id=d2cb8addb97b421fa9b5a240d24daf46]) successfully deleted
I20240501 07:58:59.799687 16612 ts_tablet_manager.cc:1875] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:58:59.800917 16614 ts_tablet_manager.cc:1888] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.801021 16614 log.cc:1196] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/84abf83338e84bbf92a1b41741003845
I20240501 07:58:59.801350 16614 ts_tablet_manager.cc:1909] T 84abf83338e84bbf92a1b41741003845 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:58:59.801359 16568 raft_consensus.cc:2781] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.801455 16568 raft_consensus.cc:482] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.801504 16568 raft_consensus.cc:3037] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.802457 16568 raft_consensus.cc:504] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.802807 16568 leader_election.cc:288] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:58:59.803421 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5805339d2c48699ece42341cd34025" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:58:59.803524 9666 raft_consensus.cc:3037] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.804428 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table_new_name [id=d2cb8addb97b421fa9b5a240d24daf46]) successfully deleted
I20240501 07:58:59.804438 9666 raft_consensus.cc:2454] T 1e5805339d2c48699ece42341cd34025 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:59.804632 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5805339d2c48699ece42341cd34025" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:58:59.804738 7272 raft_consensus.cc:3037] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.804742 16612 ts_tablet_manager.cc:1888] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:58:59.804809 16612 log.cc:1196] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/84abf83338e84bbf92a1b41741003845
I20240501 07:58:59.805186 16612 ts_tablet_manager.cc:1909] T 84abf83338e84bbf92a1b41741003845 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:58:59.805619 7272 raft_consensus.cc:2454] T 1e5805339d2c48699ece42341cd34025 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:58:59.805840 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 84abf83338e84bbf92a1b41741003845 (table rebalancer_dynamic_table_new_name [id=d2cb8addb97b421fa9b5a240d24daf46]) successfully deleted
I20240501 07:58:59.805869 7076 leader_election.cc:302] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:59.806038 16414 raft_consensus.cc:482] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.806129 16414 raft_consensus.cc:504] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.806553 16414 leader_election.cc:288] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:59.806887 16568 raft_consensus.cc:2781] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.806921 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c374464c8fd741fc92645d35e191a644" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:58:59.807049 7140 raft_consensus.cc:2454] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:59.807135 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c374464c8fd741fc92645d35e191a644" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:59.807241 7009 raft_consensus.cc:2454] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:58:59.806986 16568 raft_consensus.cc:686] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:58:59.807528 7208 leader_election.cc:302] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:59.807579 16568 consensus_queue.cc:237] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:58:59.807888 16590 raft_consensus.cc:2781] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.807938 16590 raft_consensus.cc:482] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.807976 16590 raft_consensus.cc:3037] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.808884 16590 raft_consensus.cc:504] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.809217 16590 leader_election.cc:288] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:58:59.809542 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c374464c8fd741fc92645d35e191a644" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:58:59.809643 7140 raft_consensus.cc:3037] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.810020 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c374464c8fd741fc92645d35e191a644" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:59.810115 7009 raft_consensus.cc:3037] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.810474 6730 catalog_manager.cc:5484] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:58:59.810706 7140 raft_consensus.cc:2454] T c374464c8fd741fc92645d35e191a644 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:59.811035 7206 leader_election.cc:302] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:58:59.811218 16590 raft_consensus.cc:2781] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.811261 7009 raft_consensus.cc:2454] T c374464c8fd741fc92645d35e191a644 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:58:59.811298 16590 raft_consensus.cc:686] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:58:59.811419 16590 consensus_queue.cc:237] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:58:59.813591 6734 catalog_manager.cc:5484] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:58:59.828548 16590 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.828665 16590 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.829140 16590 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:59.829999 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:58:59.830593 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:59.831854 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:59.831975 9266 raft_consensus.cc:2379] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15ab96f6477b4291932409b80af07213 in current term 1: Already voted for candidate ac6c8bc1b74b483791e262abee7862cf in this term.
I20240501 07:58:59.832638 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:58:59.833778 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:58:59.834893 7208 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15ab96f6477b4291932409b80af07213; no voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:58:59.836663 16590 raft_consensus.cc:3037] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.837761 16590 raft_consensus.cc:2726] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:58:59.844506 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:58:59.858650 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:58:59.860942 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.874713 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:58:59.874837 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:58:59.898109 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:58:59.904110 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 03bc114bc207488db89306d19a8b492a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]), partition=
I20240501 07:58:59.904112 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 03bc114bc207488db89306d19a8b492a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]), partition=
I20240501 07:58:59.904408 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bc114bc207488db89306d19a8b492a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.904410 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bc114bc207488db89306d19a8b492a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.906438 16628 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:58:59.906636 16288 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:58:59.906662 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 03bc114bc207488db89306d19a8b492a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]), partition=
I20240501 07:58:59.906898 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bc114bc207488db89306d19a8b492a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.907897 16628 tablet_bootstrap.cc:654] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.908371 16288 tablet_bootstrap.cc:654] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.909593 16551 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:58:59.910750 16551 tablet_bootstrap.cc:654] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.910894 16628 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:58:59.910956 16628 ts_tablet_manager.cc:1367] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:59.912693 16628 raft_consensus.cc:348] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.912819 16628 raft_consensus.cc:374] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.912855 16628 raft_consensus.cc:729] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.912976 16628 consensus_queue.cc:260] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.913313 16628 ts_tablet_manager.cc:1398] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:59.913558 16551 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:58:59.913638 16551 ts_tablet_manager.cc:1367] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:59.914047 16551 raft_consensus.cc:348] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.914151 16551 raft_consensus.cc:374] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.914188 16551 raft_consensus.cc:729] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.914302 16551 consensus_queue.cc:260] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.915577 16288 tablet_bootstrap.cc:492] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:58:59.915668 16288 ts_tablet_manager.cc:1367] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:58:59.915840 16551 ts_tablet_manager.cc:1398] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:59.916133 16288 raft_consensus.cc:348] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.916244 16288 raft_consensus.cc:374] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.916281 16288 raft_consensus.cc:729] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.916396 16288 consensus_queue.cc:260] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.916674 16288 ts_tablet_manager.cc:1398] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.960263 9666 consensus_queue.cc:237] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:58:59.961971 7140 raft_consensus.cc:1261] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:58:59.962016 9600 consensus_peers.cc:485] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da18394593a24e61ba2f90356645621a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:59.962383 9535 raft_consensus.cc:1261] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.962411 16668 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:59.962774 16668 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:59.964545 15946 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:59.966238 9535 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:59.966315 7140 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:59.969448 6734 catalog_manager.cc:5484] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:58:59.977960 16468 raft_consensus.cc:482] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.978062 16468 raft_consensus.cc:504] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.978519 16468 leader_election.cc:288] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:59.978792 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bc114bc207488db89306d19a8b492a" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:58:59.978940 7009 raft_consensus.cc:2454] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:59.978962 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bc114bc207488db89306d19a8b492a" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:58:59.979125 9271 raft_consensus.cc:2454] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:58:59.979272 6815 leader_election.cc:302] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:59.979434 16468 raft_consensus.cc:2781] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.979489 16468 raft_consensus.cc:482] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.979558 16468 raft_consensus.cc:3037] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.981014 16468 raft_consensus.cc:504] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.981351 16468 leader_election.cc:288] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:58:59.981691 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bc114bc207488db89306d19a8b492a" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:58:59.981757 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bc114bc207488db89306d19a8b492a" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:58:59.981840 7009 raft_consensus.cc:3037] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.981854 9271 raft_consensus.cc:3037] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.982818 9271 raft_consensus.cc:2454] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:59.982820 7009 raft_consensus.cc:2454] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:58:59.983105 6815 leader_election.cc:302] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:58:59.983263 16468 raft_consensus.cc:2781] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.983340 16468 raft_consensus.cc:686] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:58:59.983474 16468 consensus_queue.cc:237] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:58:59.985659 6734 catalog_manager.cc:5484] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:58:59.993327 9535 consensus_queue.cc:237] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:58:59.994817 9666 raft_consensus.cc:1261] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.994911 6876 raft_consensus.cc:1261] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.995339 16689 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:58:59.995630 9471 consensus_peers.cc:485] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a926c0a1d4b44ab8467f2bbafd4f23c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:59.995647 15445 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:58:59.997463 15445 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:59.997921 9666 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:58:59.999229 6876 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:00.001538 6730 catalog_manager.cc:5484] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:00.020727 9535 consensus_queue.cc:237] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:00.020850 15445 consensus_queue.cc:1035] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.022629 9666 raft_consensus.cc:1261] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.023056 16671 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:00.023223 9403 raft_consensus.cc:1261] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.023617 16671 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:00.024446 9470 consensus_peers.cc:485] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c3d606ace12402a86ab5ce85a98cd44. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:00.028700 16689 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:00.028932 16671 consensus_queue.cc:1035] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.031348 9403 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:00.031981 6736 catalog_manager.cc:5484] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:00.033388 9666 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:00.061772 9666 consensus_queue.cc:237] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.063097 7009 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:00.063441 9602 consensus_peers.cc:485] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:00.063437 9271 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:00.063704 16589 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:00.063903 15946 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:00.065634 15946 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:00.066076 7009 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:00.067811 9271 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:00.069615 6734 catalog_manager.cc:5484] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 3 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:00.090181 6877 consensus_queue.cc:237] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.090641 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:00.092005 7010 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:00.092346 7140 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:00.092944 6814 consensus_peers.cc:485] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28f7eaacb7694552a8fa2f98a5a0c868. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:00.093025 16468 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:00.094731 16468 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:00.096254 9266 raft_consensus.cc:1261] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.096414 7009 raft_consensus.cc:1261] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.097057 15220 consensus_queue.cc:1035] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.097309 16748 consensus_queue.cc:1035] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.098080 16468 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:00.100234 7138 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:00.100831 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]). Expected version 1 got 0
I20240501 07:59:00.102355 6736 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]). Expected version 1 got 0
I20240501 07:59:00.104032 6730 catalog_manager.cc:5484] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:00.104619 16510 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.104966 16510 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:00.106602 16762 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.106982 16762 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:00.109095 6736 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]). Expected version 1 got 0
I20240501 07:59:00.109689 6734 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04]). Expected version 1 got 0
I20240501 07:59:00.111682 16756 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.112040 16756 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:00.111953 7010 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:00.119050 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04] alter complete (version 1)
I20240501 07:59:00.208258 16568 consensus_queue.cc:1035] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.221402 16568 consensus_queue.cc:1035] T 1e5805339d2c48699ece42341cd34025 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.218961 6876 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:00.223476 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:00.224335 16590 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:00.227779 7138 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:00.229463 16803 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:00.231040 16756 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:00.231408 16756 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:00.232270 16762 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:00.232546 16762 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:00.233038 16510 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:00.233307 16510 tablet.cc:1620] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:00.237769 9535 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:00.238270 16590 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:00.240764 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=69ca3e92b959442084e957b9476b8d04] alter complete (version 2)
I20240501 07:59:00.250558 16414 consensus_queue.cc:1035] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.257901 16803 consensus_queue.cc:1035] T c374464c8fd741fc92645d35e191a644 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.262609 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.274858 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.277554 9238 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:00.282954 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:00.289089 6857 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:00.293337 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:00.296109 9382 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:00.307754 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:00.317708 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:00.325543 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4c57d736ea4b06b4944bcb490b5319 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.325695 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4c57d736ea4b06b4944bcb490b5319 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.325845 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4c57d736ea4b06b4944bcb490b5319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.326020 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4c57d736ea4b06b4944bcb490b5319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.328444 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4c57d736ea4b06b4944bcb490b5319 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.328716 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4c57d736ea4b06b4944bcb490b5319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.330942 16551 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:00.331960 16551 tablet_bootstrap.cc:654] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.332600 16288 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:00.333441 16288 tablet_bootstrap.cc:654] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.334489 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6b403c22704df4931be54984ea0d75 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.334741 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6b403c22704df4931be54984ea0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.337213 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6b403c22704df4931be54984ea0d75 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.337469 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6b403c22704df4931be54984ea0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.339731 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 78e78a3c7f504121aa5e63e345b1c86f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.339989 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e78a3c7f504121aa5e63e345b1c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.340255 16823 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:00.341384 16823 tablet_bootstrap.cc:654] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.342063 16820 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:00.343129 16820 tablet_bootstrap.cc:654] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.344383 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:00.344583 16823 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:00.344676 16823 ts_tablet_manager.cc:1367] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:00.344833 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6b403c22704df4931be54984ea0d75 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.345083 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6b403c22704df4931be54984ea0d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.345153 16823 raft_consensus.cc:348] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.345270 16823 raft_consensus.cc:374] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.345304 16823 raft_consensus.cc:729] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.345422 16823 consensus_queue.cc:260] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.345703 16823 ts_tablet_manager.cc:1398] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.347677 16820 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:00.347771 16820 ts_tablet_manager.cc:1367] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:00.348104 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 78e78a3c7f504121aa5e63e345b1c86f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.348234 16820 raft_consensus.cc:348] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.348335 16820 raft_consensus.cc:374] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.348371 16820 raft_consensus.cc:729] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.348376 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e78a3c7f504121aa5e63e345b1c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.348487 16820 consensus_queue.cc:260] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.350332 16820 ts_tablet_manager.cc:1398] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:00.350646 16820 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:00.351687 16820 tablet_bootstrap.cc:654] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.352852 16820 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:00.352948 16820 ts_tablet_manager.cc:1367] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:00.353432 16820 raft_consensus.cc:348] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.353533 16820 raft_consensus.cc:374] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.353561 16820 raft_consensus.cc:729] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.353659 16820 consensus_queue.cc:260] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.353952 16820 ts_tablet_manager.cc:1398] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.359992 16756 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:00.361480 16762 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:00.362152 16510 tablet.cc:1722] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:00.363250 16413 raft_consensus.cc:482] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.363348 16413 raft_consensus.cc:504] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.363737 16413 leader_election.cc:288] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:00.364902 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
W20240501 07:59:00.365803 9337 leader_election.cc:341] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:00.367389 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:00.367516 7272 raft_consensus.cc:2454] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:00.367805 9337 leader_election.cc:302] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters: fc946354b5d04503a56122e59fc320be
I20240501 07:59:00.368185 15181 raft_consensus.cc:2781] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.368263 15181 raft_consensus.cc:482] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.368314 15181 raft_consensus.cc:3037] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.369391 15181 raft_consensus.cc:504] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.369748 15181 leader_election.cc:288] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:00.371471 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:00.371626 7272 raft_consensus.cc:3037] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.372591 7272 raft_consensus.cc:2454] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:00.372848 9337 leader_election.cc:302] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:00.373025 15181 raft_consensus.cc:2781] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.373091 15181 raft_consensus.cc:686] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:00.373031 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:00.373224 15181 consensus_queue.cc:237] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:59:00.373629 9337 leader_election.cc:341] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:00.374373 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 78e78a3c7f504121aa5e63e345b1c86f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=1bb86c00717b43b798a817a67786e8a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.374663 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e78a3c7f504121aa5e63e345b1c86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.375501 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=69ca3e92b959442084e957b9476b8d04] alter complete (version 3)
I20240501 07:59:00.376793 6728 catalog_manager.cc:5484] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:00.378053 16288 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:00.378163 16288 ts_tablet_manager.cc:1367] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240501 07:59:00.378198 16824 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:00.378679 16288 raft_consensus.cc:348] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.378790 16288 raft_consensus.cc:374] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.378819 16288 raft_consensus.cc:729] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.378954 16288 consensus_queue.cc:260] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.379300 16288 ts_tablet_manager.cc:1398] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.380570 16551 tablet_bootstrap.cc:492] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:00.380652 16551 ts_tablet_manager.cc:1367] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240501 07:59:00.381078 16551 raft_consensus.cc:348] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.381183 16551 raft_consensus.cc:374] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.381213 16551 raft_consensus.cc:729] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.381323 16551 consensus_queue.cc:260] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.381578 16551 ts_tablet_manager.cc:1398] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.382447 16551 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:00.383620 16551 tablet_bootstrap.cc:654] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.390131 16827 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:00.390941 16551 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:00.391058 16551 ts_tablet_manager.cc:1367] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:00.391224 16827 tablet_bootstrap.cc:654] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.391757 16824 tablet_bootstrap.cc:654] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.392195 16551 raft_consensus.cc:348] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.392344 16551 raft_consensus.cc:374] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.392401 16551 raft_consensus.cc:729] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.392535 16551 consensus_queue.cc:260] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.392954 16551 ts_tablet_manager.cc:1398] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:00.393575 16827 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:00.393996 16827 ts_tablet_manager.cc:1367] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:00.394759 16827 raft_consensus.cc:348] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.394938 16827 raft_consensus.cc:374] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.395056 16827 raft_consensus.cc:729] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.395670 16827 consensus_queue.cc:260] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.396095 16827 ts_tablet_manager.cc:1398] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.398320 16824 tablet_bootstrap.cc:492] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:00.398689 16824 ts_tablet_manager.cc:1367] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:59:00.399583 16824 raft_consensus.cc:348] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.399735 16824 raft_consensus.cc:374] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.399787 16824 raft_consensus.cc:729] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.399950 16824 consensus_queue.cc:260] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.400286 16824 ts_tablet_manager.cc:1398] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:00.400415 15181 raft_consensus.cc:482] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.400534 15181 raft_consensus.cc:504] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.400619 16824 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:00.401022 15181 leader_election.cc:288] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:00.401510 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4c57d736ea4b06b4944bcb490b5319" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:00.401556 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4c57d736ea4b06b4944bcb490b5319" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:00.401643 7009 raft_consensus.cc:2454] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:00.401661 9271 raft_consensus.cc:2454] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:00.401918 9338 leader_election.cc:302] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:00.402087 15181 raft_consensus.cc:2781] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.402151 15181 raft_consensus.cc:482] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.402189 15181 raft_consensus.cc:3037] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.402884 16824 tablet_bootstrap.cc:654] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.403431 15181 raft_consensus.cc:504] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.403772 15181 leader_election.cc:288] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:00.404114 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4c57d736ea4b06b4944bcb490b5319" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:00.404232 9271 raft_consensus.cc:3037] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.405267 9271 raft_consensus.cc:2454] T 3b4c57d736ea4b06b4944bcb490b5319 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:00.405447 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4c57d736ea4b06b4944bcb490b5319" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:00.405544 9338 leader_election.cc:302] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:00.405563 7009 raft_consensus.cc:3037] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.405810 16413 raft_consensus.cc:2781] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.405900 16413 raft_consensus.cc:686] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:00.406034 16413 consensus_queue.cc:237] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.406656 7009 raft_consensus.cc:2454] T 3b4c57d736ea4b06b4944bcb490b5319 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:00.408541 6733 catalog_manager.cc:5484] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:00.420020 16824 tablet_bootstrap.cc:492] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:00.420156 16824 ts_tablet_manager.cc:1367] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20240501 07:59:00.420766 16824 raft_consensus.cc:348] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.420895 16824 raft_consensus.cc:374] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.420928 16824 raft_consensus.cc:729] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.421074 16824 consensus_queue.cc:260] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.421442 16824 ts_tablet_manager.cc:1398] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.439841 16846 ts_tablet_manager.cc:902] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:00.440583 16846 tablet_copy_client.cc:308] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:00.441038 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet da18394593a24e61ba2f90356645621a from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:37717)
I20240501 07:59:00.441156 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet da18394593a24e61ba2f90356645621a from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:37717: session id = 15ab96f6477b4291932409b80af07213-da18394593a24e61ba2f90356645621a
I20240501 07:59:00.442116 9686 tablet_copy_source_session.cc:205] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:00.443073 16846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da18394593a24e61ba2f90356645621a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.445711 16846 tablet_copy_client.cc:783] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:00.446419 16846 tablet_copy_client.cc:647] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:00.448064 16846 tablet_copy_client.cc:523] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:00.448817 16855 ts_tablet_manager.cc:902] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:00.449371 16855 tablet_copy_client.cc:308] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:00.449505 16846 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:00.449890 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 8c3d606ace12402a86ab5ce85a98cd44 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:50509)
I20240501 07:59:00.450001 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 8c3d606ace12402a86ab5ce85a98cd44 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:50509: session id = 1a9029bee6a144938f68b363e4cca162-8c3d606ace12402a86ab5ce85a98cd44
I20240501 07:59:00.451068 9555 tablet_copy_source_session.cc:205] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:00.451972 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d606ace12402a86ab5ce85a98cd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.454259 16855 tablet_copy_client.cc:783] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:00.454322 16846 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:00.454416 16855 tablet_copy_client.cc:647] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:00.454553 16846 tablet_bootstrap.cc:492] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:59:00.454715 16846 ts_tablet_manager.cc:1367] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:00.455559 16846 raft_consensus.cc:348] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:00.455687 16846 raft_consensus.cc:729] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.455761 16855 tablet_copy_client.cc:523] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:00.455864 16846 consensus_queue.cc:260] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:00.456482 16846 ts_tablet_manager.cc:1398] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.457017 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-da18394593a24e61ba2f90356645621a received from {username='slave'} at 127.0.99.132:37717
I20240501 07:59:00.457095 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 15ab96f6477b4291932409b80af07213-da18394593a24e61ba2f90356645621a on tablet da18394593a24e61ba2f90356645621a with peer 15ab96f6477b4291932409b80af07213
I20240501 07:59:00.457438 16855 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:00.461027 16855 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:00.461278 16855 tablet_bootstrap.cc:492] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:00.461433 16855 ts_tablet_manager.cc:1367] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:00.461839 16855 raft_consensus.cc:348] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:00.461943 16855 raft_consensus.cc:729] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.462080 16855 consensus_queue.cc:260] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:00.462391 16855 ts_tablet_manager.cc:1398] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:00.463100 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-8c3d606ace12402a86ab5ce85a98cd44 received from {username='slave'} at 127.0.99.129:50509
I20240501 07:59:00.463181 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-8c3d606ace12402a86ab5ce85a98cd44 on tablet 8c3d606ace12402a86ab5ce85a98cd44 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:00.476753 16888 ts_tablet_manager.cc:902] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:00.477406 16888 tablet_copy_client.cc:308] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:59:00.477870 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 28f7eaacb7694552a8fa2f98a5a0c868 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:44257)
I20240501 07:59:00.477988 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 28f7eaacb7694552a8fa2f98a5a0c868 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:44257: session id = 7fa3a88c25964445975bb47e0feed460-28f7eaacb7694552a8fa2f98a5a0c868
I20240501 07:59:00.478948 6899 tablet_copy_source_session.cc:205] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:00.480273 16888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7eaacb7694552a8fa2f98a5a0c868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.482618 16888 tablet_copy_client.cc:783] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:00.482789 16888 tablet_copy_client.cc:647] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:00.484333 16893 ts_tablet_manager.cc:902] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:00.484633 16888 tablet_copy_client.cc:523] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:00.485878 16888 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:00.493176 16893 tablet_copy_client.cc:308] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:00.493690 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:54625)
I20240501 07:59:00.493806 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 0a926c0a1d4b44ab8467f2bbafd4f23c from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:54625: session id = 9a3888360d4d4ace96b6b909c160bfc2-0a926c0a1d4b44ab8467f2bbafd4f23c
I20240501 07:59:00.494805 9555 tablet_copy_source_session.cc:205] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:00.495797 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:00.495947 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:00.499305 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 03bc114bc207488db89306d19a8b492a
I20240501 07:59:00.499790 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 03bc114bc207488db89306d19a8b492a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:00 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:00.500074 9238 tablet_service.cc:1514] Processing DeleteTablet for tablet 03bc114bc207488db89306d19a8b492a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:00 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:00.500129 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 03bc114bc207488db89306d19a8b492a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:00 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:00.500463 16901 tablet_replica.cc:330] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:00.500666 16901 raft_consensus.cc:2229] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:00.500960 16901 raft_consensus.cc:2258] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:00.501605 16899 tablet_replica.cc:330] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:00.501842 16899 raft_consensus.cc:2229] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:00.502008 16893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.502187 16900 tablet_replica.cc:330] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:00.502027 16899 raft_consensus.cc:2258] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:00.503194 16901 ts_tablet_manager.cc:1875] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:00.503129 16899 ts_tablet_manager.cc:1875] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:00.506392 16901 ts_tablet_manager.cc:1888] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:00.506567 16901 log.cc:1196] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/03bc114bc207488db89306d19a8b492a
I20240501 07:59:00.507058 16901 ts_tablet_manager.cc:1909] T 03bc114bc207488db89306d19a8b492a P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:59:00.508057 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table_new_name [id=69ca3e92b959442084e957b9476b8d04]) successfully deleted
I20240501 07:59:00.508729 16900 raft_consensus.cc:2229] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:00.509016 16900 raft_consensus.cc:2258] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:00.509716 16899 ts_tablet_manager.cc:1888] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:00.510654 16899 log.cc:1196] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/03bc114bc207488db89306d19a8b492a
I20240501 07:59:00.510543 16900 ts_tablet_manager.cc:1875] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:00.511879 16899 ts_tablet_manager.cc:1909] T 03bc114bc207488db89306d19a8b492a P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:00.512656 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table_new_name [id=69ca3e92b959442084e957b9476b8d04]) successfully deleted
I20240501 07:59:00.516554 16893 tablet_copy_client.cc:783] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:00.516845 16893 tablet_copy_client.cc:647] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:00.518187 16900 ts_tablet_manager.cc:1888] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:00.518260 16900 log.cc:1196] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/03bc114bc207488db89306d19a8b492a
I20240501 07:59:00.518538 16900 ts_tablet_manager.cc:1909] T 03bc114bc207488db89306d19a8b492a P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:00.518690 16902 ts_tablet_manager.cc:902] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:00.519304 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 03bc114bc207488db89306d19a8b492a (table rebalancer_dynamic_table_new_name [id=69ca3e92b959442084e957b9476b8d04]) successfully deleted
I20240501 07:59:00.519546 16902 tablet_copy_client.cc:308] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:00.520179 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 4f3e4cd27e4545baa255a01023cf1d68 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:48759)
I20240501 07:59:00.520282 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 4f3e4cd27e4545baa255a01023cf1d68 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:48759: session id = ac6c8bc1b74b483791e262abee7862cf-4f3e4cd27e4545baa255a01023cf1d68
I20240501 07:59:00.521216 9686 tablet_copy_source_session.cc:205] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:00.533219 16902 ts_tablet_manager.cc:1875] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 07:59:00.541649 16893 tablet_copy_client.cc:523] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:00.543193 16893 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:00.543330 16902 ts_tablet_manager.cc:1888] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20240501 07:59:00.543530 16902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3e4cd27e4545baa255a01023cf1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.549037 16902 tablet_copy_client.cc:783] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:00.549247 16902 tablet_copy_client.cc:647] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:00.553102 16888 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:00.553418 16888 tablet_bootstrap.cc:492] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:59:00.553591 16888 ts_tablet_manager.cc:1367] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.068s user 0.027s sys 0.005s
I20240501 07:59:00.554021 16888 raft_consensus.cc:348] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.554150 16888 raft_consensus.cc:729] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.554280 16888 consensus_queue.cc:260] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.556038 16893 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:00.556315 16893 tablet_bootstrap.cc:492] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:00.556464 16893 ts_tablet_manager.cc:1367] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 07:59:00.556952 16893 raft_consensus.cc:348] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.557087 16893 raft_consensus.cc:729] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.557231 16893 consensus_queue.cc:260] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:00.557535 16893 ts_tablet_manager.cc:1398] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.558653 16888 ts_tablet_manager.cc:1398] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:00.559183 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-28f7eaacb7694552a8fa2f98a5a0c868 received from {username='slave'} at 127.0.99.136:44257
I20240501 07:59:00.559265 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-28f7eaacb7694552a8fa2f98a5a0c868 on tablet 28f7eaacb7694552a8fa2f98a5a0c868 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:00.559873 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-0a926c0a1d4b44ab8467f2bbafd4f23c received from {username='slave'} at 127.0.99.130:54625
I20240501 07:59:00.559964 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-0a926c0a1d4b44ab8467f2bbafd4f23c on tablet 0a926c0a1d4b44ab8467f2bbafd4f23c with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:00.563449 16902 tablet_copy_client.cc:523] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:00.565016 16902 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:00.567070 16059 raft_consensus.cc:482] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.567207 16059 raft_consensus.cc:504] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:00.567643 16059 leader_election.cc:288] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:00.568302 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:00.568315 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6b403c22704df4931be54984ea0d75" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:00.568862 7272 raft_consensus.cc:2379] T 5d6b403c22704df4931be54984ea0d75 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc946354b5d04503a56122e59fc320be in current term 1: Already voted for candidate fa6964452854415a85ad72f8639768d6 in this term.
I20240501 07:59:00.569213 7075 leader_election.cc:302] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6
I20240501 07:59:00.569368 16059 raft_consensus.cc:3037] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.570410 16059 raft_consensus.cc:2726] T 5d6b403c22704df4931be54984ea0d75 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:00.584003 16902 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:00.584393 16902 tablet_bootstrap.cc:492] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:59:00.584636 16902 ts_tablet_manager.cc:1367] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240501 07:59:00.585165 16902 raft_consensus.cc:348] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.585353 16902 raft_consensus.cc:729] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.585513 16902 consensus_queue.cc:260] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.586437 16902 ts_tablet_manager.cc:1398] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.586899 9686 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-4f3e4cd27e4545baa255a01023cf1d68 received from {username='slave'} at 127.0.99.135:48759
I20240501 07:59:00.586982 9686 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-4f3e4cd27e4545baa255a01023cf1d68 on tablet 4f3e4cd27e4545baa255a01023cf1d68 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:00.602306 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:00.607501 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 08cf15b5e1d24020bdf001f078074532 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]), partition=
I20240501 07:59:00.607798 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf15b5e1d24020bdf001f078074532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.607785 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 08cf15b5e1d24020bdf001f078074532 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]), partition=
I20240501 07:59:00.607932 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 08cf15b5e1d24020bdf001f078074532 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]), partition=
I20240501 07:59:00.608052 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf15b5e1d24020bdf001f078074532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.608152 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08cf15b5e1d24020bdf001f078074532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.609930 16288 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:00.610788 16288 tablet_bootstrap.cc:654] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.611363 16820 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:00.611887 16288 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:00.611974 16288 ts_tablet_manager.cc:1367] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.612308 16820 tablet_bootstrap.cc:654] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.612452 16288 raft_consensus.cc:348] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.612592 16288 raft_consensus.cc:374] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.612638 16288 raft_consensus.cc:729] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.612762 16288 consensus_queue.cc:260] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.613076 16288 ts_tablet_manager.cc:1398] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.613884 16820 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:00.613960 16820 ts_tablet_manager.cc:1367] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:00.614374 16820 raft_consensus.cc:348] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.614464 16820 raft_consensus.cc:374] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.614487 16820 raft_consensus.cc:729] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.614593 16820 consensus_queue.cc:260] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.614895 16820 ts_tablet_manager.cc:1398] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.616397 16909 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:00.617692 16909 tablet_bootstrap.cc:654] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.619103 16909 tablet_bootstrap.cc:492] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:00.619200 16909 ts_tablet_manager.cc:1367] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:00.619658 16909 raft_consensus.cc:348] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.619762 16909 raft_consensus.cc:374] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.619794 16909 raft_consensus.cc:729] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.619931 16909 consensus_queue.cc:260] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.620198 16909 ts_tablet_manager.cc:1398] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.676750 16436 raft_consensus.cc:482] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.676950 16436 raft_consensus.cc:504] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.677431 16436 leader_election.cc:288] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:00.677748 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e78a3c7f504121aa5e63e345b1c86f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:00.677927 7138 raft_consensus.cc:2454] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:00.677906 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e78a3c7f504121aa5e63e345b1c86f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:00.678030 9666 raft_consensus.cc:2454] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:00.678216 9205 leader_election.cc:302] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:00.678412 16151 raft_consensus.cc:2781] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.678468 16151 raft_consensus.cc:482] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.678503 16151 raft_consensus.cc:3037] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.679754 16151 raft_consensus.cc:504] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.680119 16151 leader_election.cc:288] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:00.680469 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e78a3c7f504121aa5e63e345b1c86f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:00.680495 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e78a3c7f504121aa5e63e345b1c86f" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:00.680593 9666 raft_consensus.cc:3037] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.680603 7138 raft_consensus.cc:3037] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.681557 9666 raft_consensus.cc:2454] T 78e78a3c7f504121aa5e63e345b1c86f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:00.681684 7138 raft_consensus.cc:2454] T 78e78a3c7f504121aa5e63e345b1c86f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:00.681824 9206 leader_election.cc:302] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:00.682260 16151 raft_consensus.cc:2781] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.682348 16151 raft_consensus.cc:686] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:00.682474 16151 consensus_queue.cc:237] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:00.685879 6732 catalog_manager.cc:5484] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:00.725287 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.725633 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.725944 9238 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:00.735463 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:00.745054 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:00.751437 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:00.754391 7120 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:00.756840 15181 raft_consensus.cc:482] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.756973 15181 raft_consensus.cc:504] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.757378 15181 leader_election.cc:288] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:00.757884 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf15b5e1d24020bdf001f078074532" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:00.758025 7009 raft_consensus.cc:2454] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:00.758273 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:00.758317 9339 leader_election.cc:302] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:00.758805 16413 raft_consensus.cc:2781] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.758864 16413 raft_consensus.cc:482] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.758901 16413 raft_consensus.cc:3037] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.760136 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf15b5e1d24020bdf001f078074532" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:00.760259 9535 raft_consensus.cc:2454] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:00.768419 16413 raft_consensus.cc:504] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.772121 16413 leader_election.cc:288] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:00.772420 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf15b5e1d24020bdf001f078074532" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:00.772545 7009 raft_consensus.cc:3037] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.772750 16413 consensus_queue.cc:1035] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.773597 7009 raft_consensus.cc:2454] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:00.773902 9339 leader_election.cc:302] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:00.774313 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08cf15b5e1d24020bdf001f078074532" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:00.774432 9535 raft_consensus.cc:3037] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.775475 9535 raft_consensus.cc:2454] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:00.775894 16413 raft_consensus.cc:2781] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.775978 16413 raft_consensus.cc:686] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:00.776141 16413 consensus_queue.cc:237] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:00.779467 6732 catalog_manager.cc:5484] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:00.801422 15181 consensus_queue.cc:1035] T 5d6b403c22704df4931be54984ea0d75 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.889889 7271 raft_consensus.cc:1203] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:00.891032 15181 consensus_queue.cc:1035] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.897037 16413 consensus_queue.cc:1035] T 3b4c57d736ea4b06b4944bcb490b5319 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.906783 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:00.911567 9535 raft_consensus.cc:1261] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.912016 15181 consensus_queue.cc:1035] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.912330 7009 raft_consensus.cc:1261] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.912880 15181 consensus_queue.cc:1035] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.918411 6733 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]). Expected version 1 got 0
I20240501 07:59:00.920295 6729 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]). Expected version 1 got 0
I20240501 07:59:00.922544 16980 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.922904 16980 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:00.924181 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]). Expected version 1 got 0
I20240501 07:59:00.926548 16979 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.926872 16979 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:00.934221 6735 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff]). Expected version 1 got 0
I20240501 07:59:00.936383 16991 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:00.936452 7009 raft_consensus.cc:1203] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:00.936848 16991 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:00.942101 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff] alter complete (version 1)
I20240501 07:59:00.952118 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.958442 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:00.959300 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:00.962615 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:00.963410 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:00.976707 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:00.980419 7120 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:00.981523 9238 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:01.028342 6877 raft_consensus.cc:1203] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:01.063246 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:01.069514 16991 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.069844 16991 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:01.070896 16980 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.071260 16980 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:01.073624 16979 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.073913 16979 tablet.cc:1620] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:01.078434 9665 raft_consensus.cc:1203] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:01.078661 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=2ed972e34a834829a902b72aa55b7dff] alter complete (version 2)
I20240501 07:59:01.088616 9535 raft_consensus.cc:1203] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:01.164664 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.170495 6857 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:01.175141 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.183658 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:01.191754 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:01.192004 16979 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.193044 16980 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.202301 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=2ed972e34a834829a902b72aa55b7dff] alter complete (version 3)
I20240501 07:59:01.204802 16991 tablet.cc:1722] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.206667 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:01.214843 9646 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:01.217118 7120 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:01.227250 9238 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:01.229744 16151 consensus_queue.cc:1035] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.256233 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:01.264041 6857 tablet_service.cc:1467] Processing CreateTablet for tablet d3529cae5067480c8cb3c3884d7a130f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.264387 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3529cae5067480c8cb3c3884d7a130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.264454 9238 tablet_service.cc:1467] Processing CreateTablet for tablet d3529cae5067480c8cb3c3884d7a130f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.264585 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc2d5ef5ff8476e821f936f023b85ed (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.264705 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3529cae5067480c8cb3c3884d7a130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.264873 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc2d5ef5ff8476e821f936f023b85ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.265290 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc2d5ef5ff8476e821f936f023b85ed (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.265542 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc2d5ef5ff8476e821f936f023b85ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.267113 9251 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc2d5ef5ff8476e821f936f023b85ed (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.267367 9251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc2d5ef5ff8476e821f936f023b85ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.267422 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 5a038fee3d3f48c1b12bc6286b51cd1c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.267716 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a038fee3d3f48c1b12bc6286b51cd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.269634 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 5a038fee3d3f48c1b12bc6286b51cd1c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.269901 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a038fee3d3f48c1b12bc6286b51cd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.272213 9646 tablet_service.cc:1467] Processing CreateTablet for tablet d3529cae5067480c8cb3c3884d7a130f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.272497 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3529cae5067480c8cb3c3884d7a130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.274068 17055 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:01.274472 17054 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:01.274816 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 5a038fee3d3f48c1b12bc6286b51cd1c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=23090464243142bfbf92501e5ff6dd27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.275087 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a038fee3d3f48c1b12bc6286b51cd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.279392 17056 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:01.282092 16436 consensus_queue.cc:1035] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.282395 17053 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:01.285708 17056 tablet_bootstrap.cc:654] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.288643 17054 tablet_bootstrap.cc:654] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.289014 17055 tablet_bootstrap.cc:654] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.290163 17053 tablet_bootstrap.cc:654] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.291054 17054 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:01.291154 17054 ts_tablet_manager.cc:1367] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:59:01.291487 17055 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:01.291643 17054 raft_consensus.cc:348] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.291769 17054 raft_consensus.cc:374] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.291772 17052 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:01.291797 17054 raft_consensus.cc:729] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.291908 17055 ts_tablet_manager.cc:1367] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:59:01.291913 17054 consensus_queue.cc:260] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.292476 17054 ts_tablet_manager.cc:1398] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.292483 17055 raft_consensus.cc:348] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.292604 17055 raft_consensus.cc:374] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.292635 17055 raft_consensus.cc:729] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.292702 17053 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:01.292816 17053 ts_tablet_manager.cc:1367] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 07:59:01.293303 17053 raft_consensus.cc:348] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.293402 17053 raft_consensus.cc:374] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.293437 17053 raft_consensus.cc:729] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.293726 17052 tablet_bootstrap.cc:654] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.294529 17055 consensus_queue.cc:260] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.294911 17055 ts_tablet_manager.cc:1398] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:01.295127 17052 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:01.295269 17052 ts_tablet_manager.cc:1367] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:01.295647 17053 consensus_queue.cc:260] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.296095 17053 ts_tablet_manager.cc:1398] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:01.296382 17052 raft_consensus.cc:348] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.296515 17052 raft_consensus.cc:374] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.296584 17052 raft_consensus.cc:729] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.296823 17052 consensus_queue.cc:260] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.297174 17052 ts_tablet_manager.cc:1398] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.297479 17052 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:01.296859 17053 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:01.299610 17052 tablet_bootstrap.cc:654] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.300680 17053 tablet_bootstrap.cc:654] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.302932 17053 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:01.303056 17053 ts_tablet_manager.cc:1367] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:01.303560 17053 raft_consensus.cc:348] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.303735 17053 raft_consensus.cc:374] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.303819 17053 raft_consensus.cc:729] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.304122 17053 consensus_queue.cc:260] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.304533 17053 ts_tablet_manager.cc:1398] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.304705 17053 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:01.305693 17053 tablet_bootstrap.cc:654] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.306913 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:01.307103 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:01.307722 17053 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:01.307811 17053 ts_tablet_manager.cc:1367] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:01.307963 17052 tablet_bootstrap.cc:492] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:01.308084 17052 ts_tablet_manager.cc:1367] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 07:59:01.308840 17052 raft_consensus.cc:348] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.308979 17052 raft_consensus.cc:374] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.309036 17052 raft_consensus.cc:729] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.309213 17052 consensus_queue.cc:260] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.309931 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 08cf15b5e1d24020bdf001f078074532
I20240501 07:59:01.310074 17053 raft_consensus.cc:348] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.310258 17053 raft_consensus.cc:374] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.310341 17053 raft_consensus.cc:729] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.310513 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cf15b5e1d24020bdf001f078074532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:01 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:01.310671 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cf15b5e1d24020bdf001f078074532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:01 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:01.310788 17052 ts_tablet_manager.cc:1398] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:01.310920 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 08cf15b5e1d24020bdf001f078074532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:01 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:01.311100 17053 consensus_queue.cc:260] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.311273 17069 tablet_replica.cc:330] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:01.311432 17069 raft_consensus.cc:2229] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:01.311574 17069 raft_consensus.cc:2258] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.311714 17067 tablet_replica.cc:330] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:01.311895 17067 raft_consensus.cc:2229] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:01.312481 17053 ts_tablet_manager.cc:1398] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:01.315479 17067 raft_consensus.cc:2258] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.316692 17069 ts_tablet_manager.cc:1875] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:01.317303 17067 ts_tablet_manager.cc:1875] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:01.319164 17069 ts_tablet_manager.cc:1888] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:01.319255 17069 log.cc:1196] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/08cf15b5e1d24020bdf001f078074532
I20240501 07:59:01.319646 17069 ts_tablet_manager.cc:1909] T 08cf15b5e1d24020bdf001f078074532 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:59:01.319864 17056 tablet_bootstrap.cc:492] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:01.320050 17056 ts_tablet_manager.cc:1367] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240501 07:59:01.320475 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table_new_name [id=2ed972e34a834829a902b72aa55b7dff]) successfully deleted
I20240501 07:59:01.320649 17067 ts_tablet_manager.cc:1888] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:01.320667 17056 raft_consensus.cc:348] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.320791 17056 raft_consensus.cc:374] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.320832 17056 raft_consensus.cc:729] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.320925 17068 tablet_replica.cc:330] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:01.320981 17056 consensus_queue.cc:260] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.321077 17068 raft_consensus.cc:2229] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:01.321178 17067 log.cc:1196] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/08cf15b5e1d24020bdf001f078074532
I20240501 07:59:01.321321 17056 ts_tablet_manager.cc:1398] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.321535 17056 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:01.321830 17068 raft_consensus.cc:2258] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.322233 17067 ts_tablet_manager.cc:1909] T 08cf15b5e1d24020bdf001f078074532 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:01.322605 17056 tablet_bootstrap.cc:654] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.323153 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table_new_name [id=2ed972e34a834829a902b72aa55b7dff]) successfully deleted
I20240501 07:59:01.323910 17068 ts_tablet_manager.cc:1875] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:01.323998 17056 tablet_bootstrap.cc:492] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:01.324087 17056 ts_tablet_manager.cc:1367] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:01.324507 17056 raft_consensus.cc:348] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.325537 17056 raft_consensus.cc:374] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.325608 17056 raft_consensus.cc:729] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.325778 17056 consensus_queue.cc:260] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.326211 17056 ts_tablet_manager.cc:1398] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:01.326833 17068 ts_tablet_manager.cc:1888] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:01.326946 17068 log.cc:1196] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/08cf15b5e1d24020bdf001f078074532
I20240501 07:59:01.327311 17068 ts_tablet_manager.cc:1909] T 08cf15b5e1d24020bdf001f078074532 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:01.328193 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 08cf15b5e1d24020bdf001f078074532 (table rebalancer_dynamic_table_new_name [id=2ed972e34a834829a902b72aa55b7dff]) successfully deleted
I20240501 07:59:01.355415 9665 consensus_queue.cc:237] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:01.357416 7271 raft_consensus.cc:1261] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.357685 9534 raft_consensus.cc:1261] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.357919 15946 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.358151 17037 consensus_queue.cc:1035] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.360049 17037 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:01.360574 7271 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:01.360679 9534 raft_consensus.cc:2932] T da18394593a24e61ba2f90356645621a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:01.364125 6737 catalog_manager.cc:5484] T da18394593a24e61ba2f90356645621a P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 2 to 3, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:01.367364 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet da18394593a24e61ba2f90356645621a with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:01.371397 17090 tablet_replica.cc:330] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:01.371568 17090 raft_consensus.cc:2229] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:01.371696 17090 raft_consensus.cc:2258] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.372128 17090 ts_tablet_manager.cc:1875] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:01.375469 17090 ts_tablet_manager.cc:1888] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:01.375630 17090 log.cc:1196] T da18394593a24e61ba2f90356645621a P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/da18394593a24e61ba2f90356645621a
I20240501 07:59:01.376250 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet da18394593a24e61ba2f90356645621a (table rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]) successfully deleted
I20240501 07:59:01.386368 16436 raft_consensus.cc:482] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.386494 16436 raft_consensus.cc:504] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.386886 16436 leader_election.cc:288] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:01.387207 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a038fee3d3f48c1b12bc6286b51cd1c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:01.387341 9665 raft_consensus.cc:2454] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:01.387658 9206 leader_election.cc:302] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:01.387739 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a038fee3d3f48c1b12bc6286b51cd1c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:01.387832 16436 raft_consensus.cc:2781] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.387878 16436 raft_consensus.cc:482] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.387877 7271 raft_consensus.cc:2454] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:01.387912 16436 raft_consensus.cc:3037] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.389024 16436 raft_consensus.cc:504] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.389364 16436 leader_election.cc:288] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:01.390010 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a038fee3d3f48c1b12bc6286b51cd1c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:01.390123 7271 raft_consensus.cc:3037] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.391109 7271 raft_consensus.cc:2454] T 5a038fee3d3f48c1b12bc6286b51cd1c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:01.391218 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a038fee3d3f48c1b12bc6286b51cd1c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:01.391338 9665 raft_consensus.cc:3037] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.391467 9205 leader_election.cc:302] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:01.391674 16436 raft_consensus.cc:2781] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.391746 16436 raft_consensus.cc:686] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:01.391882 16436 consensus_queue.cc:237] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.392287 9665 raft_consensus.cc:2454] T 5a038fee3d3f48c1b12bc6286b51cd1c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:01.393651 16993 raft_consensus.cc:482] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.393755 16993 raft_consensus.cc:504] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.394097 16993 leader_election.cc:288] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:01.394353 6733 catalog_manager.cc:5484] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:01.394678 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3529cae5067480c8cb3c3884d7a130f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:01.394789 9266 raft_consensus.cc:2454] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:01.395138 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3529cae5067480c8cb3c3884d7a130f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:01.395246 9665 raft_consensus.cc:2454] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:01.395526 6814 leader_election.cc:302] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:01.395735 16993 raft_consensus.cc:2781] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.395788 16993 raft_consensus.cc:482] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.395823 16993 raft_consensus.cc:3037] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.396970 16993 raft_consensus.cc:504] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.397338 16993 leader_election.cc:288] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:01.398039 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3529cae5067480c8cb3c3884d7a130f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:01.398156 9266 raft_consensus.cc:3037] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.399116 9266 raft_consensus.cc:2454] T d3529cae5067480c8cb3c3884d7a130f P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:01.399466 6814 leader_election.cc:302] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:01.399798 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3529cae5067480c8cb3c3884d7a130f" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:01.399902 9665 raft_consensus.cc:3037] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.400804 9665 raft_consensus.cc:2454] T d3529cae5067480c8cb3c3884d7a130f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:01.401199 16993 raft_consensus.cc:2781] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.401281 16993 raft_consensus.cc:686] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:01.401402 16993 consensus_queue.cc:237] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:01.403988 6733 catalog_manager.cc:5484] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:01.414826 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:01.422071 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60]), partition=
I20240501 07:59:01.422087 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60]), partition=
I20240501 07:59:01.422252 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60]), partition=
I20240501 07:59:01.422353 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.422356 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.422478 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.424371 17114 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:01.424551 17052 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:01.425012 17115 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:01.425561 17052 tablet_bootstrap.cc:654] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.425561 17114 tablet_bootstrap.cc:654] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.425776 17115 tablet_bootstrap.cc:654] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.426739 17052 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:01.426831 17052 ts_tablet_manager.cc:1367] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:01.427224 17115 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:01.427330 17052 raft_consensus.cc:348] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.427453 17052 raft_consensus.cc:374] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.427486 17052 raft_consensus.cc:729] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.427614 17052 consensus_queue.cc:260] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.427305 17115 ts_tablet_manager.cc:1367] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:01.427986 17052 ts_tablet_manager.cc:1398] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.428984 17115 raft_consensus.cc:348] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.429193 17114 tablet_bootstrap.cc:492] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:01.429291 17114 ts_tablet_manager.cc:1367] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:01.429291 17115 raft_consensus.cc:374] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.429608 17115 raft_consensus.cc:729] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.429785 17115 consensus_queue.cc:260] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.429742 17114 raft_consensus.cc:348] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.430091 17114 raft_consensus.cc:374] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.430200 17114 raft_consensus.cc:729] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.430305 17115 ts_tablet_manager.cc:1398] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:01.430413 17114 consensus_queue.cc:260] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.430831 17114 ts_tablet_manager.cc:1398] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.435379 16754 raft_consensus.cc:482] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.435492 16754 raft_consensus.cc:504] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.435889 16754 leader_election.cc:288] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:01.436554 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc2d5ef5ff8476e821f936f023b85ed" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:01.436697 9266 raft_consensus.cc:2454] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:01.436807 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc2d5ef5ff8476e821f936f023b85ed" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:01.436946 9401 raft_consensus.cc:2454] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:01.437080 6814 leader_election.cc:302] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:01.437269 16754 raft_consensus.cc:2781] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.437335 16754 raft_consensus.cc:482] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.437373 16754 raft_consensus.cc:3037] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.438547 16754 raft_consensus.cc:504] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.438925 16754 leader_election.cc:288] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:01.439416 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc2d5ef5ff8476e821f936f023b85ed" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:01.439536 9266 raft_consensus.cc:3037] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.440543 9266 raft_consensus.cc:2454] T 2dc2d5ef5ff8476e821f936f023b85ed P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:01.440835 6814 leader_election.cc:302] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:01.441030 16754 raft_consensus.cc:2781] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.441097 16754 raft_consensus.cc:686] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:01.441236 16754 consensus_queue.cc:237] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.443328 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc2d5ef5ff8476e821f936f023b85ed" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:01.443454 9401 raft_consensus.cc:3037] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.444324 6729 catalog_manager.cc:5484] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:01.444492 9401 raft_consensus.cc:2454] T 2dc2d5ef5ff8476e821f936f023b85ed P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:01.543565 16059 raft_consensus.cc:482] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.543705 16059 raft_consensus.cc:504] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.544123 16059 leader_election.cc:288] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:01.544492 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0caf641d084b7a8eb7eaf8871bb0a0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:01.544636 9534 raft_consensus.cc:2454] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:01.545034 7077 leader_election.cc:302] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:01.545208 16059 raft_consensus.cc:2781] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.545267 16059 raft_consensus.cc:482] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.545317 16059 raft_consensus.cc:3037] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.546483 16059 raft_consensus.cc:504] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.546864 16059 leader_election.cc:288] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:01.547232 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0caf641d084b7a8eb7eaf8871bb0a0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:01.547291 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0caf641d084b7a8eb7eaf8871bb0a0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:01.547356 9534 raft_consensus.cc:3037] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.547413 6876 raft_consensus.cc:3037] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.548374 9534 raft_consensus.cc:2454] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:01.548460 6876 raft_consensus.cc:2454] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:01.548811 7077 leader_election.cc:302] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:01.549049 16059 raft_consensus.cc:2781] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.549152 16059 raft_consensus.cc:686] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:01.549305 16059 consensus_queue.cc:237] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:01.551141 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0caf641d084b7a8eb7eaf8871bb0a0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:01.551296 6877 raft_consensus.cc:2362] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1. Re-sending same reply.
I20240501 07:59:01.551635 6729 catalog_manager.cc:5484] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:01.578450 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:01.579136 6858 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:01.582942 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.584584 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.584592 7120 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.587455 9644 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:01.593071 9250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.634086 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:01.667435 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:01.672690 6877 raft_consensus.cc:1261] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.672804 9534 raft_consensus.cc:1261] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.673175 16568 consensus_queue.cc:1035] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.673327 16059 consensus_queue.cc:1035] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.677209 6732 catalog_manager.cc:5576] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209) does not have the latest schema for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (table rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60]). Expected version 1 got 0
I20240501 07:59:01.677124 17171 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:01.678050 17171 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:01.678510 6729 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (table rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60]). Expected version 1 got 0
I20240501 07:59:01.678910 16980 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:01.679261 16980 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:01.680418 17166 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:01.683748 17166 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:01.683967 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60] alter complete (version 1)
I20240501 07:59:01.686369 9534 consensus_queue.cc:237] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:01.688266 7010 raft_consensus.cc:1261] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.688264 6877 raft_consensus.cc:1261] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.688696 17050 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.689182 17050 consensus_queue.cc:1035] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.690793 17049 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:01.692005 7010 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:01.692536 6877 raft_consensus.cc:2932] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:01.696606 6729 catalog_manager.cc:5484] T 0a926c0a1d4b44ab8467f2bbafd4f23c P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:01.699512 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a926c0a1d4b44ab8467f2bbafd4f23c with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:01.699954 17181 tablet_replica.cc:330] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:01.700096 17181 raft_consensus.cc:2229] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:01.700253 17181 raft_consensus.cc:2258] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.700675 17181 ts_tablet_manager.cc:1875] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:01.702450 17181 ts_tablet_manager.cc:1888] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:01.702522 17181 log.cc:1196] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/0a926c0a1d4b44ab8467f2bbafd4f23c
I20240501 07:59:01.703081 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 0a926c0a1d4b44ab8467f2bbafd4f23c (table rebalancer_extra_table_22 [id=e7c0b41904474ad7a867dd5037af0309]) successfully deleted
I20240501 07:59:01.773576 16436 consensus_queue.cc:1035] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.779117 16151 consensus_queue.cc:1035] T 5a038fee3d3f48c1b12bc6286b51cd1c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.786856 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:01.792815 17166 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.793155 17166 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:01.793675 17171 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.794008 17171 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:01.797044 16980 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:01.797402 16980 tablet.cc:1620] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:01.805053 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=2f9688f36c7a4af78dedc2a9e0e3ae60] alter complete (version 2)
I20240501 07:59:01.821713 16993 consensus_queue.cc:1035] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.832787 16993 consensus_queue.cc:1035] T d3529cae5067480c8cb3c3884d7a130f P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.847443 6858 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:01.854486 6989 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.855854 9644 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:01.858825 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.859136 9515 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:01.864019 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:01.866402 9250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.888059 17230 consensus_queue.cc:1035] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.895534 17230 consensus_queue.cc:1035] T 2dc2d5ef5ff8476e821f936f023b85ed P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.906646 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:01.913662 17166 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.914309 7120 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:01.918802 17171 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.926821 16980 tablet.cc:1722] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:01.928210 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=2f9688f36c7a4af78dedc2a9e0e3ae60] alter complete (version 3)
I20240501 07:59:01.963891 9266 consensus_queue.cc:237] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:01.965293 9401 raft_consensus.cc:1261] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.965663 16151 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.965931 9534 raft_consensus.cc:1261] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:01.966305 16151 consensus_queue.cc:1035] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:01.967581 16151 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:01.967916 9401 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:01.969671 9534 raft_consensus.cc:2932] T 6a550051551640dcab9733948bb0966f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:01.970983 6730 catalog_manager.cc:5484] T 6a550051551640dcab9733948bb0966f P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:01.975358 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a550051551640dcab9733948bb0966f with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:01.975725 17263 tablet_replica.cc:330] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:01.975919 17263 raft_consensus.cc:2229] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:01.976101 17263 raft_consensus.cc:2258] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:01.976647 17263 ts_tablet_manager.cc:1875] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:01.980849 17263 ts_tablet_manager.cc:1888] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:01.980993 17263 log.cc:1196] T 6a550051551640dcab9733948bb0966f P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a550051551640dcab9733948bb0966f
I20240501 07:59:01.980899 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:01.981628 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 6a550051551640dcab9733948bb0966f (table rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]) successfully deleted
I20240501 07:59:01.988260 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 743c2dc9554a41b4ae093ae2205cc835 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.988584 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743c2dc9554a41b4ae093ae2205cc835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.988749 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 743c2dc9554a41b4ae093ae2205cc835 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.989005 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743c2dc9554a41b4ae093ae2205cc835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.989333 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 941785ea6d6d459ebdce786b805d2a7a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.989557 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941785ea6d6d459ebdce786b805d2a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.989595 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 941785ea6d6d459ebdce786b805d2a7a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.989847 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941785ea6d6d459ebdce786b805d2a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.991194 9644 tablet_service.cc:1467] Processing CreateTablet for tablet be4268a9b2cb4d229a8b26d12371f2e2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.991449 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4268a9b2cb4d229a8b26d12371f2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.992372 7251 tablet_service.cc:1467] Processing CreateTablet for tablet be4268a9b2cb4d229a8b26d12371f2e2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.992655 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4268a9b2cb4d229a8b26d12371f2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.993722 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 941785ea6d6d459ebdce786b805d2a7a (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.993935 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941785ea6d6d459ebdce786b805d2a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.994132 17271 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:01.994980 17274 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:01.995242 17271 tablet_bootstrap.cc:654] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.995995 17274 tablet_bootstrap.cc:654] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.996282 17273 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:01.997011 17274 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:01.997092 17274 ts_tablet_manager.cc:1367] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:01.997529 17273 tablet_bootstrap.cc:654] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.997498 17274 raft_consensus.cc:348] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:01.997601 17274 raft_consensus.cc:374] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.997630 17274 raft_consensus.cc:729] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.997737 17274 consensus_queue.cc:260] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:01.997990 17274 ts_tablet_manager.cc:1398] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.998315 17272 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:01.999634 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 743c2dc9554a41b4ae093ae2205cc835 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.999912 17272 tablet_bootstrap.cc:654] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.999931 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743c2dc9554a41b4ae093ae2205cc835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.000346 17270 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:02.001008 17273 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:02.001111 17273 ts_tablet_manager.cc:1367] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:01.999634 9381 tablet_service.cc:1467] Processing CreateTablet for tablet be4268a9b2cb4d229a8b26d12371f2e2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=9dfa6a5401284dfc891e79b9287ffc30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.001319 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4268a9b2cb4d229a8b26d12371f2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.001399 17270 tablet_bootstrap.cc:654] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.001544 17273 raft_consensus.cc:348] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.001646 17273 raft_consensus.cc:374] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.001674 17273 raft_consensus.cc:729] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.001783 17273 consensus_queue.cc:260] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.002472 17281 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:02.003221 17273 ts_tablet_manager.cc:1398] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:02.003443 17273 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:02.003654 17271 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:02.003728 17281 tablet_bootstrap.cc:654] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.003757 17271 ts_tablet_manager.cc:1367] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:02.004241 17271 raft_consensus.cc:348] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.004494 17273 tablet_bootstrap.cc:654] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.004372 17271 raft_consensus.cc:374] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.004963 17281 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:02.005054 17281 ts_tablet_manager.cc:1367] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.005347 17271 raft_consensus.cc:729] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.005489 17281 raft_consensus.cc:348] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.005519 17271 consensus_queue.cc:260] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.005600 17281 raft_consensus.cc:374] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.005636 17281 raft_consensus.cc:729] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.005636 17272 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:02.005720 17272 ts_tablet_manager.cc:1367] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:02.005749 17281 consensus_queue.cc:260] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.005997 17281 ts_tablet_manager.cc:1398] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.006145 17281 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:02.006227 17272 raft_consensus.cc:348] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.006323 17272 raft_consensus.cc:374] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.006352 17272 raft_consensus.cc:729] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.006453 17272 consensus_queue.cc:260] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.006695 17272 ts_tablet_manager.cc:1398] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:02.006940 17281 tablet_bootstrap.cc:654] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.008157 17273 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:02.008225 17273 ts_tablet_manager.cc:1367] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:02.008651 17273 raft_consensus.cc:348] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.008762 17273 raft_consensus.cc:374] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.008795 17273 raft_consensus.cc:729] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.008916 17273 consensus_queue.cc:260] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.009198 17273 ts_tablet_manager.cc:1398] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.010452 17271 ts_tablet_manager.cc:1398] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:02.012413 17281 tablet_bootstrap.cc:492] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:02.012787 17281 ts_tablet_manager.cc:1367] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 07:59:02.013933 17281 raft_consensus.cc:348] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.014104 17281 raft_consensus.cc:374] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.014178 17281 raft_consensus.cc:729] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.014444 17281 consensus_queue.cc:260] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.014609 17270 tablet_bootstrap.cc:492] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:02.014724 17270 ts_tablet_manager.cc:1367] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:02.015103 17281 ts_tablet_manager.cc:1398] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:02.015201 17270 raft_consensus.cc:348] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.015313 17270 raft_consensus.cc:374] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.015347 17270 raft_consensus.cc:729] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.015453 17270 consensus_queue.cc:260] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.015717 17270 ts_tablet_manager.cc:1398] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.015913 17270 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:02.016832 17270 tablet_bootstrap.cc:654] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.020277 17270 tablet_bootstrap.cc:492] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:02.020357 17270 ts_tablet_manager.cc:1367] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:02.020743 17270 raft_consensus.cc:348] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.020838 17270 raft_consensus.cc:374] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.020864 17270 raft_consensus.cc:729] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.020979 17270 consensus_queue.cc:260] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.021241 17270 ts_tablet_manager.cc:1398] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:02.030436 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:02.030593 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:02.033119 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 6d0caf641d084b7a8eb7eaf8871bb0a0
I20240501 07:59:02.033650 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:02.033737 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:02.033762 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:02.034067 17291 tablet_replica.cc:330] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:02.034313 17291 raft_consensus.cc:2229] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.034664 17291 raft_consensus.cc:2258] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.034170 17292 tablet_replica.cc:330] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:02.035310 17292 raft_consensus.cc:2229] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:02.035424 17291 ts_tablet_manager.cc:1875] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.035688 17292 raft_consensus.cc:2258] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.036310 17292 ts_tablet_manager.cc:1875] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.038712 17292 ts_tablet_manager.cc:1888] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.038841 17292 log.cc:1196] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6d0caf641d084b7a8eb7eaf8871bb0a0
I20240501 07:59:02.039216 17292 ts_tablet_manager.cc:1909] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:59:02.039512 17291 ts_tablet_manager.cc:1888] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.039589 17291 log.cc:1196] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/6d0caf641d084b7a8eb7eaf8871bb0a0
I20240501 07:59:02.039887 17291 ts_tablet_manager.cc:1909] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:59:02.040069 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (table rebalancer_dynamic_table_new_name [id=2f9688f36c7a4af78dedc2a9e0e3ae60]) successfully deleted
I20240501 07:59:02.040547 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (table rebalancer_dynamic_table_new_name [id=2f9688f36c7a4af78dedc2a9e0e3ae60]) successfully deleted
I20240501 07:59:02.043159 17293 tablet_replica.cc:330] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:02.043319 17293 raft_consensus.cc:2229] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.043471 17293 raft_consensus.cc:2258] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.044013 17293 ts_tablet_manager.cc:1875] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.046238 17293 ts_tablet_manager.cc:1888] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.046326 17293 log.cc:1196] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d0caf641d084b7a8eb7eaf8871bb0a0
I20240501 07:59:02.046654 17293 ts_tablet_manager.cc:1909] T 6d0caf641d084b7a8eb7eaf8871bb0a0 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:59:02.051411 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 6d0caf641d084b7a8eb7eaf8871bb0a0 (table rebalancer_dynamic_table_new_name [id=2f9688f36c7a4af78dedc2a9e0e3ae60]) successfully deleted
I20240501 07:59:02.080197 14116 raft_consensus.cc:482] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.080333 14116 raft_consensus.cc:504] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.080828 14116 leader_election.cc:288] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:02.081180 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:02.081346 7138 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:02.081621 9469 leader_election.cc:302] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:02.082445 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:02.082598 9666 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:02.083122 14116 raft_consensus.cc:2781] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.083200 14116 raft_consensus.cc:482] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.083245 14116 raft_consensus.cc:3037] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.084399 14116 raft_consensus.cc:504] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.084738 14116 leader_election.cc:288] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:02.085315 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:02.085438 7138 raft_consensus.cc:3037] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.086477 7138 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:02.086776 9469 leader_election.cc:302] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:02.086859 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:02.086969 14116 raft_consensus.cc:2781] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.086979 9666 raft_consensus.cc:3037] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.087134 14116 raft_consensus.cc:686] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:02.087270 14116 consensus_queue.cc:237] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:02.088050 9666 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:02.089996 6735 catalog_manager.cc:5484] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:02.135061 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:02.141103 6989 tablet_service.cc:1467] Processing CreateTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]), partition=
I20240501 07:59:02.141162 9250 tablet_service.cc:1467] Processing CreateTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]), partition=
I20240501 07:59:02.141337 6858 tablet_service.cc:1467] Processing CreateTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]), partition=
I20240501 07:59:02.141393 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cc1e1b4e9a4990ababdfeef9429c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.141402 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cc1e1b4e9a4990ababdfeef9429c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.141572 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cc1e1b4e9a4990ababdfeef9429c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.143926 17271 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:02.144836 17271 tablet_bootstrap.cc:654] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.146036 17271 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:02.146059 17321 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:02.146131 17271 ts_tablet_manager.cc:1367] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:02.146625 17271 raft_consensus.cc:348] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.146739 17271 raft_consensus.cc:374] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.146771 17271 raft_consensus.cc:729] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.146893 17271 consensus_queue.cc:260] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.147063 17321 tablet_bootstrap.cc:654] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.147104 17320 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:02.147210 17271 ts_tablet_manager.cc:1398] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:02.148990 17321 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:02.149070 17321 ts_tablet_manager.cc:1367] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.149468 17321 raft_consensus.cc:348] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.149559 17321 raft_consensus.cc:374] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.149585 17321 raft_consensus.cc:729] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.149705 17321 consensus_queue.cc:260] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.149945 17321 ts_tablet_manager.cc:1398] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:02.150014 17320 tablet_bootstrap.cc:654] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.151208 17320 tablet_bootstrap.cc:492] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:02.151295 17320 ts_tablet_manager.cc:1367] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:02.151718 17320 raft_consensus.cc:348] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.151871 17320 raft_consensus.cc:374] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.151953 17320 raft_consensus.cc:729] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.152101 17320 consensus_queue.cc:260] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.152897 17320 ts_tablet_manager.cc:1398] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:02.243353 17230 raft_consensus.cc:482] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.243484 17230 raft_consensus.cc:504] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.243989 17230 leader_election.cc:288] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:02.244383 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cc1e1b4e9a4990ababdfeef9429c36" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:02.244457 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cc1e1b4e9a4990ababdfeef9429c36" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:02.244544 9266 raft_consensus.cc:2454] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:02.244606 7010 raft_consensus.cc:2454] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:02.244807 6814 leader_election.cc:302] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:02.245043 17230 raft_consensus.cc:2781] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.245107 17230 raft_consensus.cc:482] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.245162 17230 raft_consensus.cc:3037] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.246158 17230 raft_consensus.cc:504] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.246467 17230 leader_election.cc:288] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:02.246821 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cc1e1b4e9a4990ababdfeef9429c36" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:02.246821 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cc1e1b4e9a4990ababdfeef9429c36" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:02.246939 9266 raft_consensus.cc:3037] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.246939 7010 raft_consensus.cc:3037] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.247927 7010 raft_consensus.cc:2454] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:02.247927 9266 raft_consensus.cc:2454] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:02.248208 6815 leader_election.cc:302] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:02.249652 17230 raft_consensus.cc:2781] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.249742 17230 raft_consensus.cc:686] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:02.249882 17230 consensus_queue.cc:237] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.252360 6735 catalog_manager.cc:5484] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:02.294013 16569 raft_consensus.cc:482] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.294131 16569 raft_consensus.cc:504] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.294592 16569 leader_election.cc:288] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:02.294917 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743c2dc9554a41b4ae093ae2205cc835" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:02.295034 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743c2dc9554a41b4ae093ae2205cc835" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:02.295087 7138 raft_consensus.cc:2454] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:02.295183 9403 raft_consensus.cc:2454] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:02.295370 6944 leader_election.cc:302] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:02.295528 16569 raft_consensus.cc:2781] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.295583 16569 raft_consensus.cc:482] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.295616 16569 raft_consensus.cc:3037] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.296777 16569 raft_consensus.cc:504] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.297082 16569 leader_election.cc:288] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:02.297355 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743c2dc9554a41b4ae093ae2205cc835" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:02.297402 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743c2dc9554a41b4ae093ae2205cc835" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:02.297466 7140 raft_consensus.cc:3037] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.297520 9403 raft_consensus.cc:3037] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.298357 7140 raft_consensus.cc:2454] T 743c2dc9554a41b4ae093ae2205cc835 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:02.298425 9403 raft_consensus.cc:2454] T 743c2dc9554a41b4ae093ae2205cc835 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:02.298768 6945 leader_election.cc:302] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:02.298947 16569 raft_consensus.cc:2781] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.299064 16569 raft_consensus.cc:686] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:02.299198 16569 consensus_queue.cc:237] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:02.301676 6735 catalog_manager.cc:5484] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:02.313022 15946 raft_consensus.cc:482] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.313148 15946 raft_consensus.cc:504] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.313573 15946 leader_election.cc:288] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:02.314045 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4268a9b2cb4d229a8b26d12371f2e2" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:02.314085 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4268a9b2cb4d229a8b26d12371f2e2" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:02.314196 7271 raft_consensus.cc:2454] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:02.314213 9403 raft_consensus.cc:2454] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:02.314512 9600 leader_election.cc:302] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:02.314683 15946 raft_consensus.cc:2781] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.314785 15946 raft_consensus.cc:482] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.314834 15946 raft_consensus.cc:3037] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.315722 15946 raft_consensus.cc:504] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.316051 15946 leader_election.cc:288] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:02.316365 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4268a9b2cb4d229a8b26d12371f2e2" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:02.316406 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4268a9b2cb4d229a8b26d12371f2e2" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:02.316466 9403 raft_consensus.cc:3037] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.316534 7271 raft_consensus.cc:3037] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.317220 9403 raft_consensus.cc:2454] T be4268a9b2cb4d229a8b26d12371f2e2 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:02.317382 7271 raft_consensus.cc:2454] T be4268a9b2cb4d229a8b26d12371f2e2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:02.317494 9601 leader_election.cc:302] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:02.317657 15946 raft_consensus.cc:2781] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.317823 15946 raft_consensus.cc:686] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:02.317963 15946 consensus_queue.cc:237] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.320567 6735 catalog_manager.cc:5484] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:02.344485 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.344938 6858 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.351439 9250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.351786 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:02.352455 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.352937 9515 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 07:59:02.383216 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.385994 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:02.391171 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:02.393568 9266 raft_consensus.cc:1261] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.393975 7010 raft_consensus.cc:1261] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.394059 17229 consensus_queue.cc:1035] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.394552 17229 consensus_queue.cc:1035] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.398062 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]). Expected version 1 got 0
I20240501 07:59:02.398649 6735 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]). Expected version 1 got 0
I20240501 07:59:02.406212 17171 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:02.406582 17171 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:02.411854 6731 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]). Expected version 1 got 0
I20240501 07:59:02.412321 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1] alter complete (version 1)
I20240501 07:59:02.436807 17339 mvcc.cc:205] Tried to move back new op lower bound from 7022798202438877184 to 7022798201856876544. Current Snapshot: MvccSnapshot[applied={T|T < 7022798202438877184}]
I20240501 07:59:02.437078 17343 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:02.437422 17343 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:02.438021 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1]). Expected version 1 got 0
I20240501 07:59:02.451321 17344 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:02.455921 17344 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:02.479327 9403 consensus_queue.cc:237] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.480789 6877 raft_consensus.cc:1261] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:02.480922 9534 raft_consensus.cc:1261] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:02.481398 16413 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:02.481609 17005 consensus_queue.cc:1035] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:02.483562 16413 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:02.486470 6877 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:02.487164 6735 catalog_manager.cc:5484] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:02.487382 9534 raft_consensus.cc:2932] T 79ab7f97eb2e439f8c974636b5791824 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:02.490486 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ab7f97eb2e439f8c974636b5791824 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:02.490921 17369 tablet_replica.cc:330] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:02.491122 17369 raft_consensus.cc:2229] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.491261 17369 raft_consensus.cc:2258] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.491717 17369 ts_tablet_manager.cc:1875] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:02.494174 17369 ts_tablet_manager.cc:1888] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:02.494256 17369 log.cc:1196] T 79ab7f97eb2e439f8c974636b5791824 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/79ab7f97eb2e439f8c974636b5791824
I20240501 07:59:02.494781 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 79ab7f97eb2e439f8c974636b5791824 (table rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]) successfully deleted
I20240501 07:59:02.517678 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:02.523567 17171 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:02.523913 17171 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:02.524003 17344 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:02.524266 17344 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:02.527446 17343 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:02.527799 17343 tablet.cc:1620] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:02.529757 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=0f88b324ebd44f7da734a7ef01ab30d1] alter complete (version 2)
I20240501 07:59:02.586822 14116 consensus_queue.cc:1035] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.593410 16671 consensus_queue.cc:1035] T 941785ea6d6d459ebdce786b805d2a7a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.632349 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:02.640525 17171 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:02.642519 17343 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:02.642611 17344 tablet.cc:1722] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:02.652056 6858 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.653398 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=0f88b324ebd44f7da734a7ef01ab30d1] alter complete (version 3)
I20240501 07:59:02.654415 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.656826 9250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.657090 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.659679 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:02.659674 9515 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 07:59:02.671272 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:02.689975 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.719244 15946 consensus_queue.cc:1035] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.730320 15946 consensus_queue.cc:1035] T be4268a9b2cb4d229a8b26d12371f2e2 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.748508 17277 consensus_queue.cc:1035] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.754505 16569 consensus_queue.cc:1035] T 743c2dc9554a41b4ae093ae2205cc835 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.758278 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:02.758468 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:02.761238 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet c1cc1e1b4e9a4990ababdfeef9429c36
I20240501 07:59:02.761754 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:02.762035 17445 tablet_replica.cc:330] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:02.762163 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:02.762423 17446 tablet_replica.cc:330] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:02.762503 17445 raft_consensus.cc:2229] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.762665 17446 raft_consensus.cc:2229] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.762696 17445 raft_consensus.cc:2258] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.762727 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cc1e1b4e9a4990ababdfeef9429c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:02 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:02.762850 17446 raft_consensus.cc:2258] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.763028 17447 tablet_replica.cc:330] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:02.763311 17447 raft_consensus.cc:2229] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:02.763625 17447 raft_consensus.cc:2258] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.764029 17445 ts_tablet_manager.cc:1875] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.764045 17446 ts_tablet_manager.cc:1875] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.764241 17447 ts_tablet_manager.cc:1875] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:02.766177 17447 ts_tablet_manager.cc:1888] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.766247 17447 log.cc:1196] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c1cc1e1b4e9a4990ababdfeef9429c36
I20240501 07:59:02.766326 17445 ts_tablet_manager.cc:1888] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.766327 17446 ts_tablet_manager.cc:1888] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:02.766405 17446 log.cc:1196] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c1cc1e1b4e9a4990ababdfeef9429c36
I20240501 07:59:02.766405 17445 log.cc:1196] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c1cc1e1b4e9a4990ababdfeef9429c36
I20240501 07:59:02.766573 17447 ts_tablet_manager.cc:1909] T c1cc1e1b4e9a4990ababdfeef9429c36 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:59:02.766693 17446 ts_tablet_manager.cc:1909] T c1cc1e1b4e9a4990ababdfeef9429c36 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:02.767082 17445 ts_tablet_manager.cc:1909] T c1cc1e1b4e9a4990ababdfeef9429c36 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:02.767454 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table_new_name [id=0f88b324ebd44f7da734a7ef01ab30d1]) successfully deleted
I20240501 07:59:02.768047 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table_new_name [id=0f88b324ebd44f7da734a7ef01ab30d1]) successfully deleted
I20240501 07:59:02.769384 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet c1cc1e1b4e9a4990ababdfeef9429c36 (table rebalancer_dynamic_table_new_name [id=0f88b324ebd44f7da734a7ef01ab30d1]) successfully deleted
I20240501 07:59:02.769655 9266 consensus_queue.cc:237] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.770961 7270 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:02.771036 9403 raft_consensus.cc:1261] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:02.771420 17257 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:02.771591 17449 consensus_queue.cc:1035] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:02.773528 17449 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:02.773839 7270 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:02.773936 9403 raft_consensus.cc:2932] T 3c5e1de6fc394fcb98c64165d74ca317 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:02.777477 6736 catalog_manager.cc:5484] T 3c5e1de6fc394fcb98c64165d74ca317 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 4 to 5, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:02.781499 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5e1de6fc394fcb98c64165d74ca317 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:02.782114 17456 tablet_replica.cc:330] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:02.783206 17456 raft_consensus.cc:2229] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:02.783375 17456 raft_consensus.cc:2258] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:02.783824 17456 ts_tablet_manager.cc:1875] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:02.787253 17456 ts_tablet_manager.cc:1888] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:02.787410 17456 log.cc:1196] T 3c5e1de6fc394fcb98c64165d74ca317 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3c5e1de6fc394fcb98c64165d74ca317
I20240501 07:59:02.788422 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 3c5e1de6fc394fcb98c64165d74ca317 (table rebalancer_extra_table_10 [id=ec8cd84c027549079e88fc5284f3749b]) successfully deleted
I20240501 07:59:02.864346 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:02.871549 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]), partition=
I20240501 07:59:02.871865 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ed8f5cefc64949aca7bfb2fd79e270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.875123 17494 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:02.875563 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]), partition=
I20240501 07:59:02.875586 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]), partition=
I20240501 07:59:02.875818 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ed8f5cefc64949aca7bfb2fd79e270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.875864 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ed8f5cefc64949aca7bfb2fd79e270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.876415 17494 tablet_bootstrap.cc:654] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.877775 17494 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:02.877871 17494 ts_tablet_manager.cc:1367] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.878381 17494 raft_consensus.cc:348] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.878495 17494 raft_consensus.cc:374] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.878535 17494 raft_consensus.cc:729] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.878530 17497 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:02.878654 17494 consensus_queue.cc:260] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.878973 17494 ts_tablet_manager.cc:1398] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.878994 17496 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:02.879449 17497 tablet_bootstrap.cc:654] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.880873 17496 tablet_bootstrap.cc:654] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.881947 17496 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:02.882045 17496 ts_tablet_manager.cc:1367] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:02.882524 17496 raft_consensus.cc:348] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.882637 17496 raft_consensus.cc:374] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.882671 17496 raft_consensus.cc:729] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.882795 17496 consensus_queue.cc:260] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.883129 17496 ts_tablet_manager.cc:1398] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.885567 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:02.888566 17497 tablet_bootstrap.cc:492] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:02.889024 17497 ts_tablet_manager.cc:1367] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:02.889545 17497 raft_consensus.cc:348] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.889683 17497 raft_consensus.cc:374] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.889724 17497 raft_consensus.cc:729] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.889858 17497 consensus_queue.cc:260] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:02.890161 17497 ts_tablet_manager.cc:1398] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.901423 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 7686c82864eb49a89981252c40136528 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.901491 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 7686c82864eb49a89981252c40136528 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.901760 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7686c82864eb49a89981252c40136528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.901793 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7686c82864eb49a89981252c40136528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.901793 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 7686c82864eb49a89981252c40136528 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.902067 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7686c82864eb49a89981252c40136528. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.903838 17497 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:02.905661 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f7e700156edc4011b658891315e4be0c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.905720 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 2573148ec92040d8ae505c25f8c2640f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.905920 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e700156edc4011b658891315e4be0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.905885 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 2573148ec92040d8ae505c25f8c2640f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.906059 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573148ec92040d8ae505c25f8c2640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.906134 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573148ec92040d8ae505c25f8c2640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.907574 9250 tablet_service.cc:1467] Processing CreateTablet for tablet f7e700156edc4011b658891315e4be0c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.907860 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e700156edc4011b658891315e4be0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.908655 17496 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:02.908804 6858 tablet_service.cc:1467] Processing CreateTablet for tablet f7e700156edc4011b658891315e4be0c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.909096 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e700156edc4011b658891315e4be0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.909713 17496 tablet_bootstrap.cc:654] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.910874 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 2573148ec92040d8ae505c25f8c2640f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=dd3d48b4a78b4cb88121f48bfe5c205d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.911156 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2573148ec92040d8ae505c25f8c2640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.911669 17497 tablet_bootstrap.cc:654] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.912014 17505 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:02.912248 17503 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:02.912287 17502 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:02.912868 17497 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:02.913015 17497 ts_tablet_manager.cc:1367] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:02.913416 17502 tablet_bootstrap.cc:654] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.913583 17503 tablet_bootstrap.cc:654] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.913596 17497 raft_consensus.cc:348] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.913856 17497 raft_consensus.cc:374] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.913946 17497 raft_consensus.cc:729] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.914149 17497 consensus_queue.cc:260] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.914384 17505 tablet_bootstrap.cc:654] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.914687 17502 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:02.914772 17502 ts_tablet_manager.cc:1367] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.914831 17497 ts_tablet_manager.cc:1398] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:02.915302 17502 raft_consensus.cc:348] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.915403 17502 raft_consensus.cc:374] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.915436 17502 raft_consensus.cc:729] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.915549 17502 consensus_queue.cc:260] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.915817 17505 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:02.915823 17502 ts_tablet_manager.cc:1398] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.915894 17505 ts_tablet_manager.cc:1367] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:02.917200 17496 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:02.917282 17496 ts_tablet_manager.cc:1367] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:02.917726 17496 raft_consensus.cc:348] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.917841 17496 raft_consensus.cc:374] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.917872 17496 raft_consensus.cc:729] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.917994 17496 consensus_queue.cc:260] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.918301 17496 ts_tablet_manager.cc:1398] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.918486 17500 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:02.921278 17505 raft_consensus.cc:348] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.921569 17505 raft_consensus.cc:374] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.921645 17505 raft_consensus.cc:729] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.921792 17505 consensus_queue.cc:260] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.923290 17500 tablet_bootstrap.cc:654] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.923611 17505 ts_tablet_manager.cc:1398] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 07:59:02.923862 17501 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:02.925089 17501 tablet_bootstrap.cc:654] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.926975 17503 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:02.927065 17500 tablet_bootstrap.cc:492] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:02.927193 17500 ts_tablet_manager.cc:1367] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:02.927289 17503 ts_tablet_manager.cc:1367] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:02.928411 17501 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:02.928540 17501 ts_tablet_manager.cc:1367] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:02.928781 17500 raft_consensus.cc:348] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.928987 17500 raft_consensus.cc:374] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.929077 17500 raft_consensus.cc:729] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.929833 17503 raft_consensus.cc:348] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.929978 17503 raft_consensus.cc:374] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.930037 17503 raft_consensus.cc:729] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.930078 17501 raft_consensus.cc:348] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.930224 17501 raft_consensus.cc:374] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.930500 17501 raft_consensus.cc:729] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.930685 17501 consensus_queue.cc:260] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.929504 17500 consensus_queue.cc:260] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:02.931034 17501 ts_tablet_manager.cc:1398] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:02.931174 17500 ts_tablet_manager.cc:1398] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:02.931988 17501 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:02.933058 17501 tablet_bootstrap.cc:654] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.933391 17503 consensus_queue.cc:260] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.934366 17500 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:02.935519 17500 tablet_bootstrap.cc:654] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.939608 17501 tablet_bootstrap.cc:492] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:02.940752 17500 tablet_bootstrap.cc:492] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:02.940840 17500 ts_tablet_manager.cc:1367] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:02.940982 17501 ts_tablet_manager.cc:1367] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:02.941339 17500 raft_consensus.cc:348] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.941459 17500 raft_consensus.cc:374] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.941495 17500 raft_consensus.cc:729] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.941627 17500 consensus_queue.cc:260] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:02.941654 17501 raft_consensus.cc:348] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.941850 17501 raft_consensus.cc:374] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.941922 17500 ts_tablet_manager.cc:1398] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:02.941919 17501 raft_consensus.cc:729] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.942119 17501 consensus_queue.cc:260] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:02.942535 17501 ts_tablet_manager.cc:1398] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.944295 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.935618 17503 ts_tablet_manager.cc:1398] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:02.951349 6858 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.958773 9250 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:02.962072 9515 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 07:59:02.969110 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:02.976326 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:02.982635 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:03.015702 7121 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:03.042071 16671 raft_consensus.cc:482] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.042198 16671 raft_consensus.cc:504] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:03.042594 16671 leader_election.cc:288] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:03.043185 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ed8f5cefc64949aca7bfb2fd79e270" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:03.043326 9403 raft_consensus.cc:2454] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:03.043479 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ed8f5cefc64949aca7bfb2fd79e270" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:03.043622 7009 raft_consensus.cc:2454] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:03.043761 9470 leader_election.cc:302] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:03.043942 16671 raft_consensus.cc:2781] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.044003 16671 raft_consensus.cc:482] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.044035 16671 raft_consensus.cc:3037] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.045101 16671 raft_consensus.cc:504] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:03.045431 16671 leader_election.cc:288] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:03.045701 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ed8f5cefc64949aca7bfb2fd79e270" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:03.045723 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ed8f5cefc64949aca7bfb2fd79e270" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:03.045820 9403 raft_consensus.cc:3037] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.045820 7009 raft_consensus.cc:3037] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.046797 7009 raft_consensus.cc:2454] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:03.046797 9403 raft_consensus.cc:2454] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:03.047101 9471 leader_election.cc:302] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:03.047336 16671 raft_consensus.cc:2781] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.047459 16671 raft_consensus.cc:686] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:03.047587 16671 consensus_queue.cc:237] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:03.050133 6733 catalog_manager.cc:5484] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:03.064690 16671 raft_consensus.cc:482] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.064807 16671 raft_consensus.cc:504] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.065214 16671 leader_election.cc:288] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:03.065486 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:03.065626 7009 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:03.065670 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:03.065814 9666 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:03.065944 9471 leader_election.cc:302] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:03.066113 16671 raft_consensus.cc:2781] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.066179 16671 raft_consensus.cc:482] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.066222 16671 raft_consensus.cc:3037] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.067145 16671 raft_consensus.cc:504] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.067555 16671 leader_election.cc:288] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:03.067776 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:03.067840 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:03.067901 7009 raft_consensus.cc:3037] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.067955 9666 raft_consensus.cc:3037] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.068773 7009 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:03.069057 9666 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:03.069128 9471 leader_election.cc:302] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:03.069476 16671 raft_consensus.cc:2781] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.069628 16671 raft_consensus.cc:686] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:03.069756 16671 consensus_queue.cc:237] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.071975 6733 catalog_manager.cc:5484] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.072984 9403 consensus_queue.cc:237] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.074208 9666 raft_consensus.cc:1261] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:03.074321 7270 raft_consensus.cc:1261] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:03.074625 17364 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:03.074873 16413 consensus_queue.cc:1035] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:03.076506 17364 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.076972 9666 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.077153 7270 raft_consensus.cc:2932] T d3333244e06a4ad8bd3ecc73cb356c90 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.080461 6737 catalog_manager.cc:5484] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.084025 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet d3333244e06a4ad8bd3ecc73cb356c90 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:03.084295 17456 tablet_replica.cc:330] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:03.084455 17456 raft_consensus.cc:2229] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:03.084589 17456 raft_consensus.cc:2258] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.085067 17456 ts_tablet_manager.cc:1875] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:03.088104 17456 ts_tablet_manager.cc:1888] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:03.088232 17456 log.cc:1196] T d3333244e06a4ad8bd3ecc73cb356c90 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d3333244e06a4ad8bd3ecc73cb356c90
I20240501 07:59:03.088753 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet d3333244e06a4ad8bd3ecc73cb356c90 (table rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]) successfully deleted
I20240501 07:59:03.103966 17257 raft_consensus.cc:482] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.104082 17257 raft_consensus.cc:504] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:03.104521 17257 leader_election.cc:288] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:03.104833 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e700156edc4011b658891315e4be0c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:03.104985 7270 raft_consensus.cc:2454] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:03.104959 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e700156edc4011b658891315e4be0c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:03.105095 6877 raft_consensus.cc:2454] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:03.105312 9205 leader_election.cc:302] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:03.105492 17257 raft_consensus.cc:2781] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.105561 17257 raft_consensus.cc:482] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.105600 17257 raft_consensus.cc:3037] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.106590 17257 raft_consensus.cc:504] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:03.106949 17257 leader_election.cc:288] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:03.107300 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e700156edc4011b658891315e4be0c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:03.107436 7270 raft_consensus.cc:3037] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.107442 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e700156edc4011b658891315e4be0c" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:03.107558 6877 raft_consensus.cc:3037] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.108266 7270 raft_consensus.cc:2454] T f7e700156edc4011b658891315e4be0c P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:03.108417 6877 raft_consensus.cc:2454] T f7e700156edc4011b658891315e4be0c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:03.108565 9205 leader_election.cc:302] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:03.108734 17257 raft_consensus.cc:2781] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.108875 17257 raft_consensus.cc:686] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:03.109014 17257 consensus_queue.cc:237] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:03.111922 6733 catalog_manager.cc:5484] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:03.143903 16059 raft_consensus.cc:482] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.144027 16059 raft_consensus.cc:504] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:03.144484 16059 leader_election.cc:288] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:03.145164 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573148ec92040d8ae505c25f8c2640f" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:03.145292 7270 raft_consensus.cc:2454] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:03.145365 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573148ec92040d8ae505c25f8c2640f" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:03.145480 9666 raft_consensus.cc:2454] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:03.145562 7075 leader_election.cc:302] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:03.145776 16059 raft_consensus.cc:2781] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.145824 16059 raft_consensus.cc:482] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.145859 16059 raft_consensus.cc:3037] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.146929 16059 raft_consensus.cc:504] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:03.147289 16059 leader_election.cc:288] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:03.147545 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573148ec92040d8ae505c25f8c2640f" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:03.147634 9666 raft_consensus.cc:3037] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.148195 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2573148ec92040d8ae505c25f8c2640f" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:03.148308 7270 raft_consensus.cc:3037] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.148607 9666 raft_consensus.cc:2454] T 2573148ec92040d8ae505c25f8c2640f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:03.149057 7076 leader_election.cc:302] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:03.149212 16059 raft_consensus.cc:2781] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.149297 7270 raft_consensus.cc:2454] T 2573148ec92040d8ae505c25f8c2640f P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:03.149346 16059 raft_consensus.cc:686] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:03.149483 16059 consensus_queue.cc:237] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:03.151755 6733 catalog_manager.cc:5484] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:03.156407 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:03.161832 7009 raft_consensus.cc:1261] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.162289 16671 consensus_queue.cc:1035] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.163178 9403 raft_consensus.cc:1261] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.163615 14116 consensus_queue.cc:1035] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.166769 6733 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]). Expected version 1 got 0
I20240501 07:59:03.168378 6735 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]). Expected version 1 got 0
I20240501 07:59:03.169979 17605 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.170507 17605 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:03.172432 6735 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]). Expected version 1 got 0
I20240501 07:59:03.181816 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295] alter complete (version 1)
I20240501 07:59:03.183816 17606 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.184298 17606 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:03.185871 6735 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295]). Expected version 1 got 0
I20240501 07:59:03.187314 17604 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.188117 17604 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:03.285153 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:03.287272 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:03.294915 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:03.298372 17604 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.298760 17604 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:03.298755 17606 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.299125 17606 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:03.301770 17605 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.302060 17605 tablet.cc:1620] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:03.304939 6858 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:03.308390 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=deb21b8177ff48e7b22147cde63ea295] alter complete (version 2)
I20240501 07:59:03.312384 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:03.314754 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:03.316300 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:03.323303 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:03.335844 7121 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:03.411233 7010 consensus_queue.cc:237] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:03.412606 9401 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:03.412740 7270 raft_consensus.cc:1261] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:03.413050 17277 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.413228 16569 consensus_queue.cc:1035] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.414232 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:03.415583 16569 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:03.416683 7270 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:03.419495 9401 raft_consensus.cc:2932] T 2d9379f9bf86403ebf1695263db9a603 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:03.422657 17605 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:03.420040 6731 catalog_manager.cc:5484] T 2d9379f9bf86403ebf1695263db9a603 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 4 to 6, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:03.423826 17604 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:03.430439 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9379f9bf86403ebf1695263db9a603 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:03.431133 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=deb21b8177ff48e7b22147cde63ea295] alter complete (version 3)
I20240501 07:59:03.431416 17671 tablet_replica.cc:330] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:03.431360 17606 tablet.cc:1722] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:03.431591 17671 raft_consensus.cc:2229] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:03.432515 17671 raft_consensus.cc:2258] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.433154 17671 ts_tablet_manager.cc:1875] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:03.435835 17671 ts_tablet_manager.cc:1888] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:59:03.435968 17671 log.cc:1196] T 2d9379f9bf86403ebf1695263db9a603 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2d9379f9bf86403ebf1695263db9a603
I20240501 07:59:03.437146 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 2d9379f9bf86403ebf1695263db9a603 (table rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]) successfully deleted
I20240501 07:59:03.537659 6730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:03.537968 6730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:03.540369 6730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 92ed8f5cefc64949aca7bfb2fd79e270
I20240501 07:59:03.541078 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:03 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:03.541349 17456 tablet_replica.cc:330] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:03.541424 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:03 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:03.541502 17456 raft_consensus.cc:2229] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:03.541677 17456 raft_consensus.cc:2258] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.541949 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ed8f5cefc64949aca7bfb2fd79e270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:03 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:03.542272 17710 tablet_replica.cc:330] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:03.542407 17710 raft_consensus.cc:2229] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:03.542542 17710 raft_consensus.cc:2258] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.543160 17709 tablet_replica.cc:330] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:03.543258 17710 ts_tablet_manager.cc:1875] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:03.543341 17709 raft_consensus.cc:2229] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:03.543473 17709 raft_consensus.cc:2258] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.543802 17456 ts_tablet_manager.cc:1875] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:03.544231 17709 ts_tablet_manager.cc:1875] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:03.546036 17456 ts_tablet_manager.cc:1888] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:03.546097 17456 log.cc:1196] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/92ed8f5cefc64949aca7bfb2fd79e270
I20240501 07:59:03.546388 17456 ts_tablet_manager.cc:1909] T 92ed8f5cefc64949aca7bfb2fd79e270 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:59:03.546749 17709 ts_tablet_manager.cc:1888] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:03.546813 17709 log.cc:1196] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/92ed8f5cefc64949aca7bfb2fd79e270
I20240501 07:59:03.547144 17709 ts_tablet_manager.cc:1909] T 92ed8f5cefc64949aca7bfb2fd79e270 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:03.547394 17710 ts_tablet_manager.cc:1888] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:03.547454 17710 log.cc:1196] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/92ed8f5cefc64949aca7bfb2fd79e270
I20240501 07:59:03.547710 17710 ts_tablet_manager.cc:1909] T 92ed8f5cefc64949aca7bfb2fd79e270 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:03.548247 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table_new_name [id=deb21b8177ff48e7b22147cde63ea295]) successfully deleted
I20240501 07:59:03.548593 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table_new_name [id=deb21b8177ff48e7b22147cde63ea295]) successfully deleted
I20240501 07:59:03.551553 17449 consensus_queue.cc:1035] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.551908 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 92ed8f5cefc64949aca7bfb2fd79e270 (table rebalancer_dynamic_table_new_name [id=deb21b8177ff48e7b22147cde63ea295]) successfully deleted
I20240501 07:59:03.572360 17257 consensus_queue.cc:1035] T f7e700156edc4011b658891315e4be0c P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.580649 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:03.586330 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:03.594419 6858 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:03.602881 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:03.603439 9646 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:03.612608 7121 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:03.615741 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:03.625267 9515 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 07:59:03.642433 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:03.648208 14116 consensus_queue.cc:1035] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.650005 6858 tablet_service.cc:1467] Processing CreateTablet for tablet f11716396dfd4d768f9d40895338d375 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]), partition=
I20240501 07:59:03.650310 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11716396dfd4d768f9d40895338d375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.650635 7121 tablet_service.cc:1467] Processing CreateTablet for tablet f11716396dfd4d768f9d40895338d375 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]), partition=
I20240501 07:59:03.650890 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11716396dfd4d768f9d40895338d375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.654541 9250 tablet_service.cc:1467] Processing CreateTablet for tablet f11716396dfd4d768f9d40895338d375 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]), partition=
I20240501 07:59:03.654812 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11716396dfd4d768f9d40895338d375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.655747 17728 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:03.657020 17728 tablet_bootstrap.cc:654] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.657711 17729 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:03.658110 17728 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:03.658202 17728 ts_tablet_manager.cc:1367] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:03.658696 17728 raft_consensus.cc:348] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.658828 17728 raft_consensus.cc:374] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.658874 17728 raft_consensus.cc:729] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.659013 17728 consensus_queue.cc:260] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.659358 17728 ts_tablet_manager.cc:1398] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.659687 17729 tablet_bootstrap.cc:654] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.661020 17729 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:03.661149 17729 ts_tablet_manager.cc:1367] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:03.661600 17729 raft_consensus.cc:348] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.661989 17729 raft_consensus.cc:374] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.662036 17729 raft_consensus.cc:729] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.662191 17729 consensus_queue.cc:260] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.662685 17729 ts_tablet_manager.cc:1398] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.666831 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:03.674661 9515 tablet_service.cc:1467] Processing CreateTablet for tablet ef5cce31b78f402dae9530fc3079c28b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.675052 6989 tablet_service.cc:1467] Processing CreateTablet for tablet ef5cce31b78f402dae9530fc3079c28b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.675127 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 75a5833b927e402aa86f218408592de2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.675360 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5cce31b78f402dae9530fc3079c28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.675400 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a5833b927e402aa86f218408592de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.675452 7251 tablet_service.cc:1467] Processing CreateTablet for tablet b56cc0ecf4014eb99ffebbf975be91ca (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.675710 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56cc0ecf4014eb99ffebbf975be91ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.675737 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5cce31b78f402dae9530fc3079c28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.677805 17729 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:03.678525 9646 tablet_service.cc:1467] Processing CreateTablet for tablet b56cc0ecf4014eb99ffebbf975be91ca (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.678856 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56cc0ecf4014eb99ffebbf975be91ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.679162 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 75a5833b927e402aa86f218408592de2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.679397 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a5833b927e402aa86f218408592de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.679715 7121 tablet_service.cc:1467] Processing CreateTablet for tablet ef5cce31b78f402dae9530fc3079c28b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.679957 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5cce31b78f402dae9530fc3079c28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.681557 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 75a5833b927e402aa86f218408592de2 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.681838 17729 tablet_bootstrap.cc:654] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.681856 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a5833b927e402aa86f218408592de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.682401 17735 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:03.682374 17727 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:03.682660 17733 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:03.683786 17727 tablet_bootstrap.cc:654] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.683919 14116 consensus_queue.cc:1035] T 7686c82864eb49a89981252c40136528 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.685169 17733 tablet_bootstrap.cc:654] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.685940 17732 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:03.686909 17728 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:03.687251 17734 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:03.688009 17728 tablet_bootstrap.cc:654] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.688376 17734 tablet_bootstrap.cc:654] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.683820 17735 tablet_bootstrap.cc:654] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.689762 17732 tablet_bootstrap.cc:654] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.691522 9515 tablet_service.cc:1467] Processing CreateTablet for tablet b56cc0ecf4014eb99ffebbf975be91ca (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=077aa1bdb07a4f01b682db269afee391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.691771 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b56cc0ecf4014eb99ffebbf975be91ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.691954 17728 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:03.692059 17728 ts_tablet_manager.cc:1367] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:03.692580 17728 raft_consensus.cc:348] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.692713 17728 raft_consensus.cc:374] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.692776 17728 raft_consensus.cc:729] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.692926 17728 consensus_queue.cc:260] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.693205 17734 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:03.693274 17728 ts_tablet_manager.cc:1398] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:03.693285 17734 ts_tablet_manager.cc:1367] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:03.693296 17732 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:03.693380 17732 ts_tablet_manager.cc:1367] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:03.694363 17732 raft_consensus.cc:348] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.694547 17732 raft_consensus.cc:374] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.694631 17732 raft_consensus.cc:729] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.694857 17732 consensus_queue.cc:260] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.695269 17732 ts_tablet_manager.cc:1398] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:03.701164 17229 raft_consensus.cc:482] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.701262 17229 raft_consensus.cc:504] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.701696 17229 leader_election.cc:288] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:03.702062 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11716396dfd4d768f9d40895338d375" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
W20240501 07:59:03.702432 6813 leader_election.cc:341] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:03.703207 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11716396dfd4d768f9d40895338d375" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:03.703373 9270 raft_consensus.cc:2454] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:03.703735 6814 leader_election.cc:302] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters: fc946354b5d04503a56122e59fc320be
I20240501 07:59:03.704218 17229 raft_consensus.cc:2781] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.704284 17229 raft_consensus.cc:482] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.704324 17229 raft_consensus.cc:3037] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.704777 17729 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:03.704885 17729 ts_tablet_manager.cc:1367] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 07:59:03.705272 17729 raft_consensus.cc:348] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.705349 17729 raft_consensus.cc:374] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.705369 17729 raft_consensus.cc:729] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.705417 17229 raft_consensus.cc:504] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.705477 17729 consensus_queue.cc:260] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.705770 17729 ts_tablet_manager.cc:1398] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:03.705781 17229 leader_election.cc:288] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:03.706779 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11716396dfd4d768f9d40895338d375" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
W20240501 07:59:03.707162 6813 leader_election.cc:341] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:03.707615 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11716396dfd4d768f9d40895338d375" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:03.707741 9270 raft_consensus.cc:3037] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.708601 17733 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:03.708695 17733 ts_tablet_manager.cc:1367] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:59:03.708782 9270 raft_consensus.cc:2454] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:03.709112 6814 leader_election.cc:302] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters: fc946354b5d04503a56122e59fc320be
I20240501 07:59:03.709218 17733 raft_consensus.cc:348] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.709326 17733 raft_consensus.cc:374] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.709357 17733 raft_consensus.cc:729] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.709354 17230 raft_consensus.cc:2781] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.709427 17230 raft_consensus.cc:686] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:03.709479 17733 consensus_queue.cc:260] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.709545 17230 consensus_queue.cc:237] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.709830 17733 ts_tablet_manager.cc:1398] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.712908 6728 catalog_manager.cc:5484] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.719081 17734 raft_consensus.cc:348] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.719357 17734 raft_consensus.cc:374] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.719422 17734 raft_consensus.cc:729] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.719571 17734 consensus_queue.cc:260] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.719923 17734 ts_tablet_manager.cc:1398] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240501 07:59:03.720220 17734 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:03.721035 17735 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:03.721423 17735 ts_tablet_manager.cc:1367] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240501 07:59:03.721495 17734 tablet_bootstrap.cc:654] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.722654 17735 raft_consensus.cc:348] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.723448 17735 raft_consensus.cc:374] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.723529 17735 raft_consensus.cc:729] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.723714 17735 consensus_queue.cc:260] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.724720 17735 ts_tablet_manager.cc:1398] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:03.726526 17734 tablet_bootstrap.cc:492] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:03.726612 17734 ts_tablet_manager.cc:1367] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:03.727200 17734 raft_consensus.cc:348] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.727301 17734 raft_consensus.cc:374] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.727327 17734 raft_consensus.cc:729] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.727418 17734 consensus_queue.cc:260] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.727654 17734 ts_tablet_manager.cc:1398] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.727805 17734 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:03.728719 17734 tablet_bootstrap.cc:654] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.729063 17727 tablet_bootstrap.cc:492] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:03.729157 17727 ts_tablet_manager.cc:1367] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20240501 07:59:03.729693 17727 raft_consensus.cc:348] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.729820 17727 raft_consensus.cc:374] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.729864 17727 raft_consensus.cc:729] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.729985 17727 consensus_queue.cc:260] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.730283 17727 ts_tablet_manager.cc:1398] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.730453 17727 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:03.731408 17727 tablet_bootstrap.cc:654] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.732682 16059 consensus_queue.cc:1035] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.735771 17727 tablet_bootstrap.cc:492] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:03.735873 17727 ts_tablet_manager.cc:1367] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:03.736500 17727 raft_consensus.cc:348] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.736620 17727 raft_consensus.cc:374] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.736657 17727 raft_consensus.cc:729] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.736769 17727 consensus_queue.cc:260] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.737052 17727 ts_tablet_manager.cc:1398] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.739354 17734 tablet_bootstrap.cc:492] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:03.739504 17734 ts_tablet_manager.cc:1367] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 07:59:03.739995 17734 raft_consensus.cc:348] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.740167 17734 raft_consensus.cc:374] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.740236 17734 raft_consensus.cc:729] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.740401 17734 consensus_queue.cc:260] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.740794 17734 ts_tablet_manager.cc:1398] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:03.743542 17598 consensus_queue.cc:1035] T 2573148ec92040d8ae505c25f8c2640f P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.768579 7271 consensus_queue.cc:237] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.769855 9535 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:03.770247 7138 raft_consensus.cc:1261] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:03.770251 17525 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.770651 16590 consensus_queue.cc:1035] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.772552 17770 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.772851 9535 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.774672 7138 raft_consensus.cc:2932] T 8c0f4ea5ccba462d9468e3f445569eaa P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.776350 6728 catalog_manager.cc:5484] T 8c0f4ea5ccba462d9468e3f445569eaa P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 4 to 6, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.779554 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0f4ea5ccba462d9468e3f445569eaa with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:03.780387 17774 tablet_replica.cc:330] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:03.780591 17774 raft_consensus.cc:2229] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:03.780758 17774 raft_consensus.cc:2258] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:03.781415 17774 ts_tablet_manager.cc:1875] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:03.792805 17774 ts_tablet_manager.cc:1888] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:59:03.792889 17774 log.cc:1196] T 8c0f4ea5ccba462d9468e3f445569eaa P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8c0f4ea5ccba462d9468e3f445569eaa
I20240501 07:59:03.793483 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 8c0f4ea5ccba462d9468e3f445569eaa (table rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]) successfully deleted
I20240501 07:59:03.807514 9665 consensus_queue.cc:237] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:03.809145 9535 raft_consensus.cc:1261] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:03.809303 9602 consensus_peers.cc:485] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d38445516d1e49d88ef9707246d70115. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.809464 9266 raft_consensus.cc:1261] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.809501 17089 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.809808 17089 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.811475 15946 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:03.811900 9266 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:03.813161 9535 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:03.815155 6733 catalog_manager.cc:5484] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:03.817168 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:03.822046 7138 raft_consensus.cc:3037] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.823292 7138 raft_consensus.cc:1261] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.823706 17229 consensus_queue.cc:1035] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.824280 9266 raft_consensus.cc:1261] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.825068 17229 consensus_queue.cc:1035] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.826581 17798 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.826993 17798 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:03.827402 6728 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]). Expected version 1 got 0
I20240501 07:59:03.830860 6730 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]). Expected version 1 got 0
I20240501 07:59:03.830862 17799 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.831192 17799 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:03.832012 17805 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:03.832445 17805 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:03.833179 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e]). Expected version 1 got 0
I20240501 07:59:03.838418 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e] alter complete (version 1)
I20240501 07:59:03.851094 7009 consensus_queue.cc:237] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:03.852689 9403 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:03.853088 7271 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:03.853287 17277 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 07:59:03.853448 6945 consensus_peers.cc:485] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 651beb63e03d40bca88f01ebed416754. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.853731 17277 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.855587 17666 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.856019 7271 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.858191 9403 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.858453 17598 raft_consensus.cc:482] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.858562 17598 raft_consensus.cc:504] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.859576 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5cce31b78f402dae9530fc3079c28b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:03.859503 6731 catalog_manager.cc:5484] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.859699 7009 raft_consensus.cc:2454] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:03.860220 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5cce31b78f402dae9530fc3079c28b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:03.860348 9535 raft_consensus.cc:2454] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:03.860725 7077 leader_election.cc:302] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:03.861089 16059 raft_consensus.cc:2781] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.861151 16059 raft_consensus.cc:482] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.861218 16059 raft_consensus.cc:3037] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.862296 16059 raft_consensus.cc:504] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.862664 16059 leader_election.cc:288] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:03.863088 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5cce31b78f402dae9530fc3079c28b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:03.863231 7009 raft_consensus.cc:3037] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.863525 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5cce31b78f402dae9530fc3079c28b" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:03.863657 9535 raft_consensus.cc:3037] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.864219 7009 raft_consensus.cc:2454] T ef5cce31b78f402dae9530fc3079c28b P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:03.864471 9535 raft_consensus.cc:2454] T ef5cce31b78f402dae9530fc3079c28b P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:03.864495 7077 leader_election.cc:302] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:03.858980 17598 leader_election.cc:288] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:03.864738 17598 raft_consensus.cc:2781] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.864807 17598 raft_consensus.cc:686] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:03.864979 17598 consensus_queue.cc:237] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.867909 6731 catalog_manager.cc:5484] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:03.885178 9665 consensus_queue.cc:237] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:03.886806 9403 raft_consensus.cc:1261] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.886842 7271 raft_consensus.cc:1261] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:03.886934 9600 consensus_peers.cc:485] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7b180ca614d4eaf812ae2cf7485b715. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.887225 15946 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.887657 15946 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.889349 17089 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:03.889815 7271 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:03.891544 9401 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:03.893589 6730 catalog_manager.cc:5484] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:03.916484 7271 consensus_queue.cc:237] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.918308 6877 raft_consensus.cc:1261] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.918308 9665 raft_consensus.cc:1261] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:03.918396 7208 consensus_peers.cc:485] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ba12580263f4bd883ad7e003ebd6c51. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.918795 17525 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.919057 16590 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:03.920969 16590 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.921602 9665 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.923069 6877 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.925809 6730 catalog_manager.cc:5484] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.940717 6728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:03.946548 17798 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.946898 17798 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: MemRowSet was empty: no flush needed.
I20240501 07:59:03.947263 17799 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.947566 17799 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:03.947916 17805 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:03.948211 17805 tablet.cc:1620] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:03.951272 7138 consensus_queue.cc:237] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
W20240501 07:59:03.953022 7076 consensus_peers.cc:485] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82c30f6292754d97a839700be93f8c07. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.953846 7009 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:03.954005 9665 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:03.954497 17598 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.954830 17598 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:03.954834 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=c33da24e0aec4061826af95faebc400e] alter complete (version 2)
I20240501 07:59:03.956936 17598 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.957438 9665 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.959118 7009 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.960623 6730 catalog_manager.cc:5484] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:03.982353 7138 consensus_queue.cc:237] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:03.982937 17229 raft_consensus.cc:482] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.983090 17229 raft_consensus.cc:504] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.983573 17229 leader_election.cc:288] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:03.983884 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a5833b927e402aa86f218408592de2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:03.983984 7271 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:03.984017 9266 raft_consensus.cc:2454] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:03.984122 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a5833b927e402aa86f218408592de2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:03.984288 9535 raft_consensus.cc:2454] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:03.984336 6814 leader_election.cc:302] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:03.984396 7009 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:03.984498 17229 raft_consensus.cc:2781] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.984537 17229 raft_consensus.cc:482] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.984573 17229 raft_consensus.cc:3037] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.984781 17598 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:03.984963 17872 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:03.985190 7076 consensus_peers.cc:485] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c977cb9c5d1b41a29e7671218b917315. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:03.985648 17229 raft_consensus.cc:504] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.985986 17229 leader_election.cc:288] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:03.986369 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a5833b927e402aa86f218408592de2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:03.986480 9535 raft_consensus.cc:3037] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.986487 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a5833b927e402aa86f218408592de2" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:03.986614 9266 raft_consensus.cc:3037] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.987334 17872 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.987604 9266 raft_consensus.cc:2454] T 75a5833b927e402aa86f218408592de2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:03.987617 9535 raft_consensus.cc:2454] T 75a5833b927e402aa86f218408592de2 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:03.987787 7271 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.987968 6815 leader_election.cc:302] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:03.988299 17796 raft_consensus.cc:2781] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.988379 17796 raft_consensus.cc:686] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:03.988521 17796 consensus_queue.cc:237] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:03.990393 7009 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:03.990923 6732 catalog_manager.cc:5484] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:03.990926 6730 catalog_manager.cc:5484] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:04.012811 15946 raft_consensus.cc:482] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.012933 15946 raft_consensus.cc:504] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:04.013363 15946 leader_election.cc:288] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:04.013695 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56cc0ecf4014eb99ffebbf975be91ca" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:04.013854 9535 raft_consensus.cc:2454] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:04.014016 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56cc0ecf4014eb99ffebbf975be91ca" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:04.014143 7271 raft_consensus.cc:2454] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:04.014149 9602 leader_election.cc:302] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:04.014400 15946 raft_consensus.cc:2781] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.014456 15946 raft_consensus.cc:482] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.014503 15946 raft_consensus.cc:3037] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.015851 15946 raft_consensus.cc:504] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:04.016216 15946 leader_election.cc:288] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:04.016892 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56cc0ecf4014eb99ffebbf975be91ca" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:04.017009 9535 raft_consensus.cc:3037] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.016937 7009 consensus_queue.cc:237] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.018085 9535 raft_consensus.cc:2454] T b56cc0ecf4014eb99ffebbf975be91ca P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:04.018568 9401 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:04.018594 7270 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:04.018834 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56cc0ecf4014eb99ffebbf975be91ca" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:04.018945 7271 raft_consensus.cc:3037] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:59:04.019069 6944 consensus_peers.cc:485] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c86818cabaf49678cfbcce3480f2645. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:04.019243 17277 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:04.019398 17666 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:04.019956 9602 leader_election.cc:302] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:04.020032 7271 raft_consensus.cc:2454] T b56cc0ecf4014eb99ffebbf975be91ca P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:04.021415 15946 raft_consensus.cc:2781] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.021513 15946 raft_consensus.cc:686] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:04.021665 15946 consensus_queue.cc:237] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:04.024098 6731 catalog_manager.cc:5484] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:04.028807 17666 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:04.029266 9401 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:04.032399 7270 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:04.033103 6730 catalog_manager.cc:5484] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 3 to 4, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:04.058750 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:04.065066 17798 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.065647 17799 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.065696 17805 tablet.cc:1722] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.072016 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=c33da24e0aec4061826af95faebc400e] alter complete (version 3)
I20240501 07:59:04.073959 7138 consensus_queue.cc:237] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.075377 7270 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 07:59:04.075462 7076 consensus_peers.cc:485] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c80723cb0ce4446483a5f1754e066081. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:04.075549 9535 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:04.075807 16059 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:04.076189 16059 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:04.078279 17598 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:04.080169 9535 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:04.081398 7270 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:04.084762 6732 catalog_manager.cc:5484] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 5 to 6, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:04.174528 6731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:04.174696 6731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:04.178510 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet f11716396dfd4d768f9d40895338d375
I20240501 07:59:04.179033 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet f11716396dfd4d768f9d40895338d375 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:04 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:04.182797 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet f11716396dfd4d768f9d40895338d375 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:04 UTC) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:04.183002 17774 tablet_replica.cc:330] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:04.183240 17774 raft_consensus.cc:2229] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:04.183372 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet f11716396dfd4d768f9d40895338d375 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:04 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:04.183470 17774 raft_consensus.cc:2258] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:04.183691 17973 tablet_replica.cc:330] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:04.183831 17973 raft_consensus.cc:2229] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:04.183969 17973 raft_consensus.cc:2258] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:04.184262 17972 tablet_replica.cc:330] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:04.184790 17972 raft_consensus.cc:2229] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:04.184965 17973 ts_tablet_manager.cc:1875] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:04.184979 17774 ts_tablet_manager.cc:1875] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:04.185006 17972 raft_consensus.cc:2258] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:04.187304 17972 ts_tablet_manager.cc:1875] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:04.190613 17774 ts_tablet_manager.cc:1888] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:04.192168 17973 ts_tablet_manager.cc:1888] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:04.192292 17973 log.cc:1196] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f11716396dfd4d768f9d40895338d375
I20240501 07:59:04.192345 17774 log.cc:1196] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f11716396dfd4d768f9d40895338d375
I20240501 07:59:04.192658 17973 ts_tablet_manager.cc:1909] T f11716396dfd4d768f9d40895338d375 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:04.192853 17774 ts_tablet_manager.cc:1909] T f11716396dfd4d768f9d40895338d375 P 1a9029bee6a144938f68b363e4cca162: Deleting consensus metadata
I20240501 07:59:04.194543 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table_new_name [id=c33da24e0aec4061826af95faebc400e]) successfully deleted
I20240501 07:59:04.195456 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table_new_name [id=c33da24e0aec4061826af95faebc400e]) successfully deleted
I20240501 07:59:04.214915 17972 ts_tablet_manager.cc:1888] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:04.218024 17972 log.cc:1196] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f11716396dfd4d768f9d40895338d375
I20240501 07:59:04.219965 17972 ts_tablet_manager.cc:1909] T f11716396dfd4d768f9d40895338d375 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:59:04.222551 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet f11716396dfd4d768f9d40895338d375 (table rebalancer_dynamic_table_new_name [id=c33da24e0aec4061826af95faebc400e]) successfully deleted
I20240501 07:59:04.237464 9515 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 07:59:04.238797 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.243444 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.243966 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.247669 6858 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.252055 7121 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.259687 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:04.267719 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.280843 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:04.287330 6989 tablet_service.cc:1467] Processing CreateTablet for tablet f0921eda5f6d4bf6a98d441290cd303e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]), partition=
I20240501 07:59:04.287428 9381 tablet_service.cc:1467] Processing CreateTablet for tablet f0921eda5f6d4bf6a98d441290cd303e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]), partition=
I20240501 07:59:04.287645 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0921eda5f6d4bf6a98d441290cd303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.287686 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0921eda5f6d4bf6a98d441290cd303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.289583 17989 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:04.289758 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f0921eda5f6d4bf6a98d441290cd303e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]), partition=
I20240501 07:59:04.289988 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0921eda5f6d4bf6a98d441290cd303e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.290473 17989 tablet_bootstrap.cc:654] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.290901 17990 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:04.291592 17989 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:04.291677 17989 ts_tablet_manager.cc:1367] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:04.291860 17990 tablet_bootstrap.cc:654] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.292181 17989 raft_consensus.cc:348] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.292287 17989 raft_consensus.cc:374] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.292318 17989 raft_consensus.cc:729] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.292429 17989 consensus_queue.cc:260] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.292727 17989 ts_tablet_manager.cc:1398] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:04.292862 17990 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:04.292949 17990 ts_tablet_manager.cc:1367] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:04.293366 17990 raft_consensus.cc:348] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.293459 17990 raft_consensus.cc:374] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.293486 17990 raft_consensus.cc:729] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.293612 17990 consensus_queue.cc:260] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.293907 17990 ts_tablet_manager.cc:1398] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.314817 17994 ts_tablet_manager.cc:902] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:04.315529 17994 tablet_copy_client.cc:308] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:59:04.315991 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 651beb63e03d40bca88f01ebed416754 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:37855)
I20240501 07:59:04.316098 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 651beb63e03d40bca88f01ebed416754 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:37855: session id = 49adb9e99822494196ed22044bfed81f-651beb63e03d40bca88f01ebed416754
I20240501 07:59:04.317111 7030 tablet_copy_source_session.cc:205] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.318183 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651beb63e03d40bca88f01ebed416754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.320873 17994 tablet_copy_client.cc:783] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.321143 17994 tablet_copy_client.cc:647] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.322928 17994 tablet_copy_client.cc:523] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.324299 17994 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:04.323858 17992 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:04.327168 17992 tablet_bootstrap.cc:654] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.329671 17992 tablet_bootstrap.cc:492] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:04.329856 17992 ts_tablet_manager.cc:1367] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:04.330446 17992 raft_consensus.cc:348] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.330763 17992 raft_consensus.cc:374] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.330857 17992 raft_consensus.cc:729] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.331115 17992 consensus_queue.cc:260] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.331585 17992 ts_tablet_manager.cc:1398] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:04.357368 17994 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.357728 17994 tablet_bootstrap.cc:492] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:59:04.357945 17994 ts_tablet_manager.cc:1367] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.000s
I20240501 07:59:04.358446 17994 raft_consensus.cc:348] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.358600 17994 raft_consensus.cc:729] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.358748 17994 consensus_queue.cc:260] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.359737 17994 ts_tablet_manager.cc:1398] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 07:59:04.360484 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-651beb63e03d40bca88f01ebed416754 received from {username='slave'} at 127.0.99.133:37855
I20240501 07:59:04.360575 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 49adb9e99822494196ed22044bfed81f-651beb63e03d40bca88f01ebed416754 on tablet 651beb63e03d40bca88f01ebed416754 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:59:04.384637 16059 consensus_queue.cc:1035] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.387442 17598 consensus_queue.cc:1035] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.393262 18021 ts_tablet_manager.cc:902] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:04.397284 18021 tablet_copy_client.cc:308] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:04.397861 18025 ts_tablet_manager.cc:902] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:04.398402 18025 tablet_copy_client.cc:308] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:04.398519 9686 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet a7b180ca614d4eaf812ae2cf7485b715 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:40629)
I20240501 07:59:04.398633 9686 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet a7b180ca614d4eaf812ae2cf7485b715 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:40629: session id = fc946354b5d04503a56122e59fc320be-a7b180ca614d4eaf812ae2cf7485b715
I20240501 07:59:04.399432 9685 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet d38445516d1e49d88ef9707246d70115 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:59385)
I20240501 07:59:04.399518 9685 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet d38445516d1e49d88ef9707246d70115 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:59385: session id = 9a3888360d4d4ace96b6b909c160bfc2-d38445516d1e49d88ef9707246d70115
I20240501 07:59:04.399601 9686 tablet_copy_source_session.cc:205] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.400444 9685 tablet_copy_source_session.cc:205] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.401546 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b180ca614d4eaf812ae2cf7485b715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.401803 17788 consensus_queue.cc:1035] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.403295 18025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38445516d1e49d88ef9707246d70115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.403930 18021 tablet_copy_client.cc:783] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.404096 18021 tablet_copy_client.cc:647] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.405097 15946 consensus_queue.cc:1035] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.416505 18025 tablet_copy_client.cc:783] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.416783 18025 tablet_copy_client.cc:647] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.417940 18025 tablet_copy_client.cc:523] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.418639 18038 ts_tablet_manager.cc:902] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:04.419157 18038 tablet_copy_client.cc:308] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:59:04.419255 18021 tablet_copy_client.cc:523] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.420174 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet c977cb9c5d1b41a29e7671218b917315 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:38089)
I20240501 07:59:04.420279 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet c977cb9c5d1b41a29e7671218b917315 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:38089: session id = 7fa3a88c25964445975bb47e0feed460-c977cb9c5d1b41a29e7671218b917315
I20240501 07:59:04.421420 7160 tablet_copy_source_session.cc:205] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.421551 18021 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:04.423599 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977cb9c5d1b41a29e7671218b917315. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.426781 18025 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:04.436053 18021 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.436321 18021 tablet_bootstrap.cc:492] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:04.436471 18021 ts_tablet_manager.cc:1367] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240501 07:59:04.436635 18038 tablet_copy_client.cc:783] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.436862 18038 tablet_copy_client.cc:647] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.437131 18021 raft_consensus.cc:348] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.437254 18021 raft_consensus.cc:729] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.437390 18021 consensus_queue.cc:260] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.437803 18021 ts_tablet_manager.cc:1398] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:04.438227 9685 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-a7b180ca614d4eaf812ae2cf7485b715 received from {username='slave'} at 127.0.99.131:40629
I20240501 07:59:04.438292 9685 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session fc946354b5d04503a56122e59fc320be-a7b180ca614d4eaf812ae2cf7485b715 on tablet a7b180ca614d4eaf812ae2cf7485b715 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:04.439126 18038 tablet_copy_client.cc:523] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.440472 18038 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:04.453392 18025 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.453701 18025 tablet_bootstrap.cc:492] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:04.453894 18025 ts_tablet_manager.cc:1367] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240501 07:59:04.454536 18025 raft_consensus.cc:348] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:04.454653 18025 raft_consensus.cc:729] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.454842 18025 consensus_queue.cc:260] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:04.456300 18025 ts_tablet_manager.cc:1398] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 07:59:04.456820 9685 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-d38445516d1e49d88ef9707246d70115 received from {username='slave'} at 127.0.99.130:59385
I20240501 07:59:04.456902 9685 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-d38445516d1e49d88ef9707246d70115 on tablet d38445516d1e49d88ef9707246d70115 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:04.479137 18038 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.479489 18038 tablet_bootstrap.cc:492] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:59:04.479717 18038 ts_tablet_manager.cc:1367] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.000s
I20240501 07:59:04.480273 18038 raft_consensus.cc:348] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:04.480410 18038 raft_consensus.cc:729] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.480557 18038 consensus_queue.cc:260] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:04.480947 18038 ts_tablet_manager.cc:1398] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:04.481738 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-c977cb9c5d1b41a29e7671218b917315 received from {username='slave'} at 127.0.99.136:38089
I20240501 07:59:04.481838 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-c977cb9c5d1b41a29e7671218b917315 on tablet c977cb9c5d1b41a29e7671218b917315 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:04.483847 17994 ts_tablet_manager.cc:902] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:04.484283 17994 tablet_copy_client.cc:308] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:59:04.484712 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet c80723cb0ce4446483a5f1754e066081 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:59:04.484817 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet c80723cb0ce4446483a5f1754e066081 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-c80723cb0ce4446483a5f1754e066081
I20240501 07:59:04.485841 7160 tablet_copy_source_session.cc:205] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.486670 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c80723cb0ce4446483a5f1754e066081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.489142 17994 tablet_copy_client.cc:783] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.489300 17994 tablet_copy_client.cc:647] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.494947 17994 tablet_copy_client.cc:523] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.498682 17994 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:04.528770 17994 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.529081 17994 tablet_bootstrap.cc:492] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:59:04.529256 17994 ts_tablet_manager.cc:1367] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.004s
I20240501 07:59:04.529687 17994 raft_consensus.cc:348] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.529815 17994 raft_consensus.cc:729] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.530002 17994 consensus_queue.cc:260] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.530357 17994 ts_tablet_manager.cc:1398] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:04.531411 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-c80723cb0ce4446483a5f1754e066081 received from {username='slave'} at 127.0.99.133:44859
I20240501 07:59:04.531505 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-c80723cb0ce4446483a5f1754e066081 on tablet c80723cb0ce4446483a5f1754e066081 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:59:04.531829 18080 ts_tablet_manager.cc:902] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:04.534018 18080 tablet_copy_client.cc:308] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:59:04.534583 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 82c30f6292754d97a839700be93f8c07 from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:44859)
I20240501 07:59:04.534679 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 82c30f6292754d97a839700be93f8c07 from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:44859: session id = 49adb9e99822494196ed22044bfed81f-82c30f6292754d97a839700be93f8c07
I20240501 07:59:04.535615 7160 tablet_copy_source_session.cc:205] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.536950 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30f6292754d97a839700be93f8c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.537117 17853 raft_consensus.cc:482] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.537232 17853 raft_consensus.cc:504] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.537745 17853 leader_election.cc:288] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:04.538242 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0921eda5f6d4bf6a98d441290cd303e" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:04.538383 7009 raft_consensus.cc:2454] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:04.539148 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0921eda5f6d4bf6a98d441290cd303e" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:04.539275 9401 raft_consensus.cc:2454] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:04.539644 18080 tablet_copy_client.cc:783] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.540160 18080 tablet_copy_client.cc:647] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.542812 7207 leader_election.cc:302] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:04.543776 17853 raft_consensus.cc:2781] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.543838 17853 raft_consensus.cc:482] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.543892 17853 raft_consensus.cc:3037] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.545017 17853 raft_consensus.cc:504] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.545416 17853 leader_election.cc:288] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:04.546607 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0921eda5f6d4bf6a98d441290cd303e" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:04.546730 7009 raft_consensus.cc:3037] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.547752 7009 raft_consensus.cc:2454] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:04.549490 7208 leader_election.cc:302] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:04.549829 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0921eda5f6d4bf6a98d441290cd303e" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:04.549944 9401 raft_consensus.cc:3037] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.550906 9401 raft_consensus.cc:2454] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:04.551932 18085 ts_tablet_manager.cc:902] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:04.552646 18085 tablet_copy_client.cc:308] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:04.552664 17853 raft_consensus.cc:2781] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.552758 17853 raft_consensus.cc:686] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:04.552904 17853 consensus_queue.cc:237] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:04.553148 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 2ba12580263f4bd883ad7e003ebd6c51 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:44297)
I20240501 07:59:04.553233 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 2ba12580263f4bd883ad7e003ebd6c51 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:44297: session id = ac6c8bc1b74b483791e262abee7862cf-2ba12580263f4bd883ad7e003ebd6c51
I20240501 07:59:04.554188 7292 tablet_copy_source_session.cc:205] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.555090 18080 tablet_copy_client.cc:523] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.556929 6731 catalog_manager.cc:5484] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:04.557386 18080 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:04.558445 18085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba12580263f4bd883ad7e003ebd6c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.560873 18085 tablet_copy_client.cc:783] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.561235 18085 tablet_copy_client.cc:647] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.563455 18085 tablet_copy_client.cc:523] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.568876 18085 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:04.570240 18021 ts_tablet_manager.cc:902] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:04.570614 18021 tablet_copy_client.cc:308] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:59:04.571539 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 0c86818cabaf49678cfbcce3480f2645 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:55099)
I20240501 07:59:04.571642 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 0c86818cabaf49678cfbcce3480f2645 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:55099: session id = fc946354b5d04503a56122e59fc320be-0c86818cabaf49678cfbcce3480f2645
I20240501 07:59:04.571786 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:04.572530 7030 tablet_copy_source_session.cc:205] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:04.573031 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.576453 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c86818cabaf49678cfbcce3480f2645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.578841 18021 tablet_copy_client.cc:783] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:04.583433 17229 consensus_queue.cc:1035] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.592002 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 3553a90852d6476ea046ba3903618cc0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.592298 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3553a90852d6476ea046ba3903618cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.578977 18021 tablet_copy_client.cc:647] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:04.592365 6858 tablet_service.cc:1467] Processing CreateTablet for tablet e8bf756d31de478aa7cba1ef071c75aa (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.592604 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bf756d31de478aa7cba1ef071c75aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.592598 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 35693d2e562f4adca5e181a44ad16e25 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.592857 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35693d2e562f4adca5e181a44ad16e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.593487 18021 tablet_copy_client.cc:523] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:04.595155 17990 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:04.595023 18021 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:04.595443 9515 tablet_service.cc:1467] Processing CreateTablet for tablet e8bf756d31de478aa7cba1ef071c75aa (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.595690 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bf756d31de478aa7cba1ef071c75aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.596161 17990 tablet_bootstrap.cc:654] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.599741 9381 tablet_service.cc:1467] Processing CreateTablet for tablet e8bf756d31de478aa7cba1ef071c75aa (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.600013 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bf756d31de478aa7cba1ef071c75aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.601771 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.602481 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 3553a90852d6476ea046ba3903618cc0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.602772 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3553a90852d6476ea046ba3903618cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.605327 17989 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:04.606281 17989 tablet_bootstrap.cc:654] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.607244 17796 consensus_queue.cc:1035] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.594962 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 3553a90852d6476ea046ba3903618cc0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.608618 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3553a90852d6476ea046ba3903618cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.612464 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 35693d2e562f4adca5e181a44ad16e25 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.612751 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35693d2e562f4adca5e181a44ad16e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.616103 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 35693d2e562f4adca5e181a44ad16e25 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=1f6cfc9ae14a411db25768aa58a2eb26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.616389 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35693d2e562f4adca5e181a44ad16e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.618098 18117 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:04.619197 18117 tablet_bootstrap.cc:654] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.619338 9515 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240501 07:59:04.623255 18117 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:04.623349 18117 ts_tablet_manager.cc:1367] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:04.623498 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:04.623792 18117 raft_consensus.cc:348] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.623926 18117 raft_consensus.cc:374] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.623956 18117 raft_consensus.cc:729] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.624064 18117 consensus_queue.cc:260] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.624348 18117 ts_tablet_manager.cc:1398] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.624393 6857 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.628386 17990 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:04.628479 17990 ts_tablet_manager.cc:1367] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240501 07:59:04.628994 17990 raft_consensus.cc:348] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.629099 17990 raft_consensus.cc:374] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.629127 17990 raft_consensus.cc:729] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.629236 17990 consensus_queue.cc:260] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.629498 17990 ts_tablet_manager.cc:1398] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.640851 18114 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:04.642244 18114 tablet_bootstrap.cc:654] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.643173 18109 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:04.645300 18021 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.645568 18021 tablet_bootstrap.cc:492] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:04.646378 18021 ts_tablet_manager.cc:1367] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240501 07:59:04.647059 18021 raft_consensus.cc:348] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.647208 18021 raft_consensus.cc:729] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.647302 18112 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:04.647357 18021 consensus_queue.cc:260] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.647701 18021 ts_tablet_manager.cc:1398] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.648170 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-0c86818cabaf49678cfbcce3480f2645 received from {username='slave'} at 127.0.99.131:55099
I20240501 07:59:04.648248 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session fc946354b5d04503a56122e59fc320be-0c86818cabaf49678cfbcce3480f2645 on tablet 0c86818cabaf49678cfbcce3480f2645 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:04.649050 18112 tablet_bootstrap.cc:654] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.650031 18110 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:04.651021 18110 tablet_bootstrap.cc:654] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.651494 17989 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:04.651592 17989 ts_tablet_manager.cc:1367] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.002s
I20240501 07:59:04.652070 17989 raft_consensus.cc:348] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.652190 17989 raft_consensus.cc:374] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.652225 17989 raft_consensus.cc:729] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.652348 17989 consensus_queue.cc:260] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.652654 17989 ts_tablet_manager.cc:1398] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.653681 18109 tablet_bootstrap.cc:654] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.656524 18085 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.656904 18085 tablet_bootstrap.cc:492] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:59:04.657120 18085 ts_tablet_manager.cc:1367] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.004s
I20240501 07:59:04.657610 18085 raft_consensus.cc:348] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.657794 18085 raft_consensus.cc:729] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.658080 18085 consensus_queue.cc:260] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.664561 18085 ts_tablet_manager.cc:1398] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 07:59:04.667768 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-2ba12580263f4bd883ad7e003ebd6c51 received from {username='slave'} at 127.0.99.135:44297
I20240501 07:59:04.667857 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-2ba12580263f4bd883ad7e003ebd6c51 on tablet 2ba12580263f4bd883ad7e003ebd6c51 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:04.667963 18114 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:04.668044 18114 ts_tablet_manager.cc:1367] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 07:59:04.668583 18114 raft_consensus.cc:348] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.667832 18110 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:04.669298 18110 ts_tablet_manager.cc:1367] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240501 07:59:04.669795 18110 raft_consensus.cc:348] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.669967 18110 raft_consensus.cc:374] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.670042 18110 raft_consensus.cc:729] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.670220 18110 consensus_queue.cc:260] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.671064 18114 raft_consensus.cc:374] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.671167 18114 raft_consensus.cc:729] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.671372 18114 consensus_queue.cc:260] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.672204 18110 ts_tablet_manager.cc:1398] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:04.672400 18110 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:04.678905 18114 ts_tablet_manager.cc:1398] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 07:59:04.682842 18109 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:04.683100 7121 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.683331 18109 ts_tablet_manager.cc:1367] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240501 07:59:04.684634 18109 raft_consensus.cc:348] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.684834 18109 raft_consensus.cc:374] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.684849 17815 raft_consensus.cc:482] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.684944 18109 raft_consensus.cc:729] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.684938 17815 raft_consensus.cc:504] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.685153 18109 consensus_queue.cc:260] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.685461 17815 leader_election.cc:288] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.685586 18109 ts_tablet_manager.cc:1398] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:04.685966 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35693d2e562f4adca5e181a44ad16e25" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:04.686086 9666 raft_consensus.cc:2454] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:04.686692 6945 leader_election.cc:302] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:04.687686 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35693d2e562f4adca5e181a44ad16e25" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:04.688181 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.688696 18110 tablet_bootstrap.cc:654] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.689767 18110 tablet_bootstrap.cc:492] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:04.689853 18110 ts_tablet_manager.cc:1367] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240501 07:59:04.690236 18110 raft_consensus.cc:348] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.690336 18110 raft_consensus.cc:374] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.690371 18110 raft_consensus.cc:729] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.690526 18110 consensus_queue.cc:260] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.691116 18110 ts_tablet_manager.cc:1398] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.691470 18110 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:04.692314 18110 tablet_bootstrap.cc:654] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.693429 6989 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 07:59:04.693780 6945 leader_election.cc:341] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:04.694842 17815 raft_consensus.cc:2781] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.694905 17815 raft_consensus.cc:482] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.694944 17815 raft_consensus.cc:3037] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.696053 17815 raft_consensus.cc:504] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.696408 17815 leader_election.cc:288] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.697062 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35693d2e562f4adca5e181a44ad16e25" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:04.697175 9665 raft_consensus.cc:3037] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.698174 9665 raft_consensus.cc:2454] T 35693d2e562f4adca5e181a44ad16e25 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:04.703392 18080 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:04.703630 18110 tablet_bootstrap.cc:492] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:04.703768 18110 ts_tablet_manager.cc:1367] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:04.703848 18080 tablet_bootstrap.cc:492] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:59:04.704080 18080 ts_tablet_manager.cc:1367] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.147s user 0.031s sys 0.000s
I20240501 07:59:04.704231 18110 raft_consensus.cc:348] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.704427 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35693d2e562f4adca5e181a44ad16e25" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:04.704636 18110 raft_consensus.cc:374] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.704711 18110 raft_consensus.cc:729] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.704859 18110 consensus_queue.cc:260] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.704886 18080 raft_consensus.cc:348] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.705003 18080 raft_consensus.cc:729] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.705056 6877 raft_consensus.cc:3037] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.705257 6945 leader_election.cc:302] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:04.705258 18080 consensus_queue.cc:260] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:04.705752 18080 ts_tablet_manager.cc:1398] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.706015 6877 raft_consensus.cc:2454] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:04.707111 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-82c30f6292754d97a839700be93f8c07 received from {username='slave'} at 127.0.99.133:44859
I20240501 07:59:04.707194 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 49adb9e99822494196ed22044bfed81f-82c30f6292754d97a839700be93f8c07 on tablet 82c30f6292754d97a839700be93f8c07 with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:59:04.707458 18110 ts_tablet_manager.cc:1398] T 35693d2e562f4adca5e181a44ad16e25 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:04.716444 17815 raft_consensus.cc:2781] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.716588 17815 raft_consensus.cc:686] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:04.716713 17815 consensus_queue.cc:237] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:04.719434 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:04.719586 6730 catalog_manager.cc:5484] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:04.725522 7009 raft_consensus.cc:1261] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.725986 17853 consensus_queue.cc:1035] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.726518 9401 raft_consensus.cc:1261] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.726994 17770 consensus_queue.cc:1035] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.730816 6730 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]). Expected version 1 got 0
I20240501 07:59:04.731025 18112 tablet_bootstrap.cc:492] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:04.731122 18112 ts_tablet_manager.cc:1367] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.000s
I20240501 07:59:04.732177 18112 raft_consensus.cc:348] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.732293 18112 raft_consensus.cc:374] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.732332 18112 raft_consensus.cc:729] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.732440 18112 consensus_queue.cc:260] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.732702 18112 ts_tablet_manager.cc:1398] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:04.734544 6733 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]). Expected version 1 got 0
I20240501 07:59:04.738809 18133 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:04.739243 18133 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:04.739269 6733 catalog_manager.cc:5576] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675) does not have the latest schema for tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]). Expected version 1 got 0
I20240501 07:59:04.739972 6729 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd]). Expected version 1 got 0
I20240501 07:59:04.740089 18131 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:04.740352 18131 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:59:04.742591 18132 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:04.743090 18132 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:04.744607 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd] alter complete (version 1)
I20240501 07:59:04.789005 14116 raft_consensus.cc:482] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.789116 14116 raft_consensus.cc:504] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.789453 14116 leader_election.cc:288] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.790016 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:04.790166 9401 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:04.790490 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:04.790550 9470 leader_election.cc:302] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:04.790625 6876 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:04.790738 16671 raft_consensus.cc:2781] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.790791 16671 raft_consensus.cc:482] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.790829 16671 raft_consensus.cc:3037] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.792014 16671 raft_consensus.cc:504] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.792364 16671 leader_election.cc:288] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.794009 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:04.794073 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:04.794162 6876 raft_consensus.cc:3037] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.794212 9401 raft_consensus.cc:3037] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.795193 6876 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:04.795214 9401 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:04.797075 9470 leader_election.cc:302] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:04.797505 16671 raft_consensus.cc:2781] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.797610 16671 raft_consensus.cc:686] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:04.797767 16671 consensus_queue.cc:237] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:04.800917 6730 catalog_manager.cc:5484] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:04.824376 17598 raft_consensus.cc:482] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.824514 17598 raft_consensus.cc:504] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.825060 17598 leader_election.cc:288] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.825580 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3553a90852d6476ea046ba3903618cc0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:04.825716 9266 raft_consensus.cc:2454] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:04.825773 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3553a90852d6476ea046ba3903618cc0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:04.825903 6876 raft_consensus.cc:2454] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:04.826174 7076 leader_election.cc:302] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:04.826364 17598 raft_consensus.cc:2781] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.826452 17598 raft_consensus.cc:482] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.826504 17598 raft_consensus.cc:3037] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.827495 17598 raft_consensus.cc:504] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.827816 17598 leader_election.cc:288] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:04.828171 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3553a90852d6476ea046ba3903618cc0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:04.828294 9266 raft_consensus.cc:3037] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.828473 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3553a90852d6476ea046ba3903618cc0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:04.828589 6876 raft_consensus.cc:3037] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.829044 9266 raft_consensus.cc:2454] T 3553a90852d6476ea046ba3903618cc0 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:04.829380 6876 raft_consensus.cc:2454] T 3553a90852d6476ea046ba3903618cc0 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:04.829413 7076 leader_election.cc:302] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:04.829581 17598 raft_consensus.cc:2781] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.829666 17598 raft_consensus.cc:686] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:04.829795 17598 consensus_queue.cc:237] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:04.832216 6733 catalog_manager.cc:5484] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:04.847466 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:04.854135 18131 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:04.854492 18131 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:59:04.854779 18132 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:04.855108 18132 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:04.855353 18133 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:04.855669 18133 tablet.cc:1620] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:04.861518 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=a58a056d9cef42889211cb2fced834dd] alter complete (version 2)
I20240501 07:59:04.913996 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:04.914417 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.915657 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:04.918681 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:04.931777 6989 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:04.931968 7138 raft_consensus.cc:1203] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:04.935079 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:04.936848 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:04.939325 9266 raft_consensus.cc:1203] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:04.952678 7009 raft_consensus.cc:1203] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:04.960357 9665 raft_consensus.cc:1203] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:04.963744 6729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:04.969846 18131 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.971344 18132 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.971638 18133 tablet.cc:1722] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:04.975409 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=a58a056d9cef42889211cb2fced834dd] alter complete (version 3)
I20240501 07:59:04.987926 6857 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.006857 9266 raft_consensus.cc:1203] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:05.046082 9266 raft_consensus.cc:1203] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:05.077085 6728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:05.077317 6728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:05.079609 6728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet f0921eda5f6d4bf6a98d441290cd303e
I20240501 07:59:05.080230 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet f0921eda5f6d4bf6a98d441290cd303e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:05.080282 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet f0921eda5f6d4bf6a98d441290cd303e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:05.080586 18230 tablet_replica.cc:330] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:05.080683 18231 tablet_replica.cc:330] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:05.080726 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet f0921eda5f6d4bf6a98d441290cd303e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:59:05.080862 18231 raft_consensus.cc:2229] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.081040 18232 tablet_replica.cc:330] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:59:05.081037 18230 raft_consensus.cc:2229] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.081065 18231 raft_consensus.cc:2258] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.081161 18230 raft_consensus.cc:2258] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.081305 18232 raft_consensus.cc:2229] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:05.081583 18232 raft_consensus.cc:2258] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.081772 18231 ts_tablet_manager.cc:1875] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.082175 18230 ts_tablet_manager.cc:1875] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.083158 18232 ts_tablet_manager.cc:1875] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.085110 18231 ts_tablet_manager.cc:1888] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.085233 18231 log.cc:1196] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0921eda5f6d4bf6a98d441290cd303e
I20240501 07:59:05.085861 18231 ts_tablet_manager.cc:1909] T f0921eda5f6d4bf6a98d441290cd303e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:05.086604 18230 ts_tablet_manager.cc:1888] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.086606 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table_new_name [id=a58a056d9cef42889211cb2fced834dd]) successfully deleted
I20240501 07:59:05.086767 18230 log.cc:1196] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f0921eda5f6d4bf6a98d441290cd303e
I20240501 07:59:05.086791 18232 ts_tablet_manager.cc:1888] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.086861 18232 log.cc:1196] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f0921eda5f6d4bf6a98d441290cd303e
I20240501 07:59:05.087090 18230 ts_tablet_manager.cc:1909] T f0921eda5f6d4bf6a98d441290cd303e P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:05.087514 18232 ts_tablet_manager.cc:1909] T f0921eda5f6d4bf6a98d441290cd303e P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:59:05.088234 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table_new_name [id=a58a056d9cef42889211cb2fced834dd]) successfully deleted
I20240501 07:59:05.088321 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet f0921eda5f6d4bf6a98d441290cd303e (table rebalancer_dynamic_table_new_name [id=a58a056d9cef42889211cb2fced834dd]) successfully deleted
I20240501 07:59:05.118399 9535 raft_consensus.cc:1203] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:05.146185 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.155691 6857 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.156327 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:05.163566 7138 raft_consensus.cc:1203] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:05.166015 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.168613 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:05.169189 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:05.179415 14116 consensus_queue.cc:1035] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.181996 6989 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:05.188562 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:05.194558 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4]), partition=
I20240501 07:59:05.194612 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4]), partition=
I20240501 07:59:05.194865 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdbbef7b1374601a6b0865ffb76385e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.194862 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdbbef7b1374601a6b0865ffb76385e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.198344 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4]), partition=
I20240501 07:59:05.198557 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdbbef7b1374601a6b0865ffb76385e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.202881 18248 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:05.203903 18247 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:05.204910 18247 tablet_bootstrap.cc:654] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.203990 18248 tablet_bootstrap.cc:654] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.206034 18248 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:05.206034 18247 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:05.206115 18248 ts_tablet_manager.cc:1367] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:05.206115 18247 ts_tablet_manager.cc:1367] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.206538 18247 raft_consensus.cc:348] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.206537 18248 raft_consensus.cc:348] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.206641 18247 raft_consensus.cc:374] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.206641 18248 raft_consensus.cc:374] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.206674 18247 raft_consensus.cc:729] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.206674 18248 raft_consensus.cc:729] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.206795 18247 consensus_queue.cc:260] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.206795 18248 consensus_queue.cc:260] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.207093 18248 ts_tablet_manager.cc:1398] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.208832 18247 ts_tablet_manager.cc:1398] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:05.213575 16671 consensus_queue.cc:1035] T e8bf756d31de478aa7cba1ef071c75aa P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.215415 18251 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:05.216485 18251 tablet_bootstrap.cc:654] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.222105 18251 tablet_bootstrap.cc:492] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:05.222191 18251 ts_tablet_manager.cc:1367] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:05.222616 18251 raft_consensus.cc:348] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.222712 18251 raft_consensus.cc:374] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.222757 18251 raft_consensus.cc:729] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.222877 18251 consensus_queue.cc:260] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.223273 18251 ts_tablet_manager.cc:1398] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.225140 17277 consensus_queue.cc:1035] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.232266 17277 consensus_queue.cc:1035] T 35693d2e562f4adca5e181a44ad16e25 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.235306 17891 consensus_queue.cc:1035] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.244052 17598 consensus_queue.cc:1035] T 3553a90852d6476ea046ba3903618cc0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.248423 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:05.312216 9666 consensus_queue.cc:237] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.313491 7138 raft_consensus.cc:1261] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:05.313556 7270 raft_consensus.cc:1261] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:05.314039 17788 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:05.314318 17788 consensus_queue.cc:1035] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:05.316201 15946 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:05.318204 7138 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:05.318761 7270 raft_consensus.cc:2932] T a7b180ca614d4eaf812ae2cf7485b715 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:05.319845 6729 catalog_manager.cc:5484] T a7b180ca614d4eaf812ae2cf7485b715 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:05.324195 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b180ca614d4eaf812ae2cf7485b715 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:05.324446 18230 tablet_replica.cc:330] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:05.324621 18230 raft_consensus.cc:2229] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.324767 18230 raft_consensus.cc:2258] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.325258 18230 ts_tablet_manager.cc:1875] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:05.327504 18230 ts_tablet_manager.cc:1888] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:05.327555 18230 log.cc:1196] T a7b180ca614d4eaf812ae2cf7485b715 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a7b180ca614d4eaf812ae2cf7485b715
I20240501 07:59:05.328148 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet a7b180ca614d4eaf812ae2cf7485b715 (table rebalancer_extra_table_26 [id=42fa8ae113754fa0833306385a1fcec5]) successfully deleted
I20240501 07:59:05.360661 18254 raft_consensus.cc:482] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.360795 18254 raft_consensus.cc:504] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.361243 18254 leader_election.cc:288] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:05.361631 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdbbef7b1374601a6b0865ffb76385e" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:05.361778 7138 raft_consensus.cc:2454] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:05.361757 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdbbef7b1374601a6b0865ffb76385e" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:05.361871 9666 raft_consensus.cc:2454] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:05.362080 9337 leader_election.cc:302] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:05.362251 18254 raft_consensus.cc:2781] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.362315 18254 raft_consensus.cc:482] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.362349 18254 raft_consensus.cc:3037] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.363487 18254 raft_consensus.cc:504] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.363796 18254 leader_election.cc:288] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:05.364079 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdbbef7b1374601a6b0865ffb76385e" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:05.364102 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdbbef7b1374601a6b0865ffb76385e" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:05.364190 9666 raft_consensus.cc:3037] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.364190 7138 raft_consensus.cc:3037] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.365134 7138 raft_consensus.cc:2454] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:05.365134 9666 raft_consensus.cc:2454] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:05.365427 9338 leader_election.cc:302] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:05.365626 18254 raft_consensus.cc:2781] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.365725 18254 raft_consensus.cc:686] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:05.365850 18254 consensus_queue.cc:237] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.368901 6732 catalog_manager.cc:5484] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:05.379053 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:05.390714 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 767acae4338b4b0dba53109bf86c13fc (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.390961 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 3c13af71c56c49c89f4ba43be9956a50 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.391098 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767acae4338b4b0dba53109bf86c13fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.390987 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 3c13af71c56c49c89f4ba43be9956a50 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.391265 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c13af71c56c49c89f4ba43be9956a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.391283 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c13af71c56c49c89f4ba43be9956a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.393651 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 767acae4338b4b0dba53109bf86c13fc (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.393915 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767acae4338b4b0dba53109bf86c13fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.395345 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 84e079a65e864d8f8fc84761766a7420 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.395587 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e079a65e864d8f8fc84761766a7420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.396152 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 3c13af71c56c49c89f4ba43be9956a50 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.396438 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c13af71c56c49c89f4ba43be9956a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.398543 18251 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:05.398694 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 84e079a65e864d8f8fc84761766a7420 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.398918 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e079a65e864d8f8fc84761766a7420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.399415 18323 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:05.399600 18251 tablet_bootstrap.cc:654] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.400553 18323 tablet_bootstrap.cc:654] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.402027 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 767acae4338b4b0dba53109bf86c13fc (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.402295 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767acae4338b4b0dba53109bf86c13fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.402909 18251 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:05.402981 18251 ts_tablet_manager.cc:1367] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:05.403499 18251 raft_consensus.cc:348] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.403601 18251 raft_consensus.cc:374] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.403637 18251 raft_consensus.cc:729] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.403764 18251 consensus_queue.cc:260] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.404043 18251 ts_tablet_manager.cc:1398] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.404631 18248 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:05.405143 18325 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:05.405045 18251 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:05.405557 18248 tablet_bootstrap.cc:654] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.406201 18325 tablet_bootstrap.cc:654] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.407311 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 84e079a65e864d8f8fc84761766a7420 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=aedba9fa7b384d1d8d728491116af282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.407615 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e079a65e864d8f8fc84761766a7420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.407743 18324 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:05.411049 18324 tablet_bootstrap.cc:654] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.412212 18248 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:05.412322 18248 ts_tablet_manager.cc:1367] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:05.412804 18248 raft_consensus.cc:348] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.412909 18248 raft_consensus.cc:374] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.412941 18248 raft_consensus.cc:729] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.413066 18248 consensus_queue.cc:260] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.413367 18248 ts_tablet_manager.cc:1398] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.415820 18251 tablet_bootstrap.cc:654] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.416368 18324 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:05.416484 18324 ts_tablet_manager.cc:1367] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:05.416952 18324 raft_consensus.cc:348] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.417062 18324 raft_consensus.cc:374] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.417094 18324 raft_consensus.cc:729] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.417225 18324 consensus_queue.cc:260] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.417508 18324 ts_tablet_manager.cc:1398] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:05.419976 18330 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:05.421133 18325 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:05.421182 18330 tablet_bootstrap.cc:654] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.421267 18325 ts_tablet_manager.cc:1367] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:05.421893 18325 raft_consensus.cc:348] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.422047 18325 raft_consensus.cc:374] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.422122 18325 raft_consensus.cc:729] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.422709 18323 tablet_bootstrap.cc:492] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:05.422855 18323 ts_tablet_manager.cc:1367] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 07:59:05.423415 18325 consensus_queue.cc:260] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.423837 18325 ts_tablet_manager.cc:1398] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:05.424316 18323 raft_consensus.cc:348] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.424492 18323 raft_consensus.cc:374] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.424566 18323 raft_consensus.cc:729] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.424762 18323 consensus_queue.cc:260] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.425176 18323 ts_tablet_manager.cc:1398] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.425590 18323 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:05.426111 18251 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:05.426198 18251 ts_tablet_manager.cc:1367] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240501 07:59:05.426591 18323 tablet_bootstrap.cc:654] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.426651 18251 raft_consensus.cc:348] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.426785 18251 raft_consensus.cc:374] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.426818 18251 raft_consensus.cc:729] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.426939 18251 consensus_queue.cc:260] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.427156 17277 raft_consensus.cc:482] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.427233 18251 ts_tablet_manager.cc:1398] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.427240 17277 raft_consensus.cc:504] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.427567 17277 leader_election.cc:288] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:05.427975 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767acae4338b4b0dba53109bf86c13fc" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
W20240501 07:59:05.428431 6946 leader_election.cc:341] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:05.430292 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767acae4338b4b0dba53109bf86c13fc" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:05.430439 7138 raft_consensus.cc:2454] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:05.431170 6944 leader_election.cc:302] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:05.431445 17815 raft_consensus.cc:2781] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.431505 17815 raft_consensus.cc:482] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.431545 17815 raft_consensus.cc:3037] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.433002 17815 raft_consensus.cc:504] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.433347 17815 leader_election.cc:288] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:05.433751 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767acae4338b4b0dba53109bf86c13fc" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
W20240501 07:59:05.434092 6946 leader_election.cc:341] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:05.434859 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767acae4338b4b0dba53109bf86c13fc" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:05.434969 7138 raft_consensus.cc:3037] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.436093 7138 raft_consensus.cc:2454] T 767acae4338b4b0dba53109bf86c13fc P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:05.436717 6944 leader_election.cc:302] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:05.436964 17815 raft_consensus.cc:2781] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.437044 17815 raft_consensus.cc:686] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:05.437168 17815 consensus_queue.cc:237] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.442167 6728 catalog_manager.cc:5484] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:05.443652 18330 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:05.443770 18330 ts_tablet_manager.cc:1367] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:05.444289 18330 raft_consensus.cc:348] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.444855 18323 tablet_bootstrap.cc:492] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:05.444962 18323 ts_tablet_manager.cc:1367] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:59:05.445463 18323 raft_consensus.cc:348] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.445581 18323 raft_consensus.cc:374] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.445622 18323 raft_consensus.cc:729] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.445756 18323 consensus_queue.cc:260] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:05.446077 18323 ts_tablet_manager.cc:1398] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:05.446164 18330 raft_consensus.cc:374] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.446259 18330 raft_consensus.cc:729] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.446286 18323 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:05.446465 18330 consensus_queue.cc:260] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.447263 18323 tablet_bootstrap.cc:654] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.448110 18330 ts_tablet_manager.cc:1398] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:05.450498 18323 tablet_bootstrap.cc:492] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:05.450600 18323 ts_tablet_manager.cc:1367] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:05.451083 18323 raft_consensus.cc:348] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.451200 18323 raft_consensus.cc:374] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.451238 18323 raft_consensus.cc:729] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.451373 18323 consensus_queue.cc:260] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.451656 18323 ts_tablet_manager.cc:1398] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.479837 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:05.480808 18290 raft_consensus.cc:482] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.480914 18290 raft_consensus.cc:504] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.481297 18290 leader_election.cc:288] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:05.481633 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c13af71c56c49c89f4ba43be9956a50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:05.481779 9401 raft_consensus.cc:2454] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:05.481916 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c13af71c56c49c89f4ba43be9956a50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:05.482059 9535 raft_consensus.cc:2454] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:05.482061 6814 leader_election.cc:302] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:05.482211 18290 raft_consensus.cc:2781] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.482275 18290 raft_consensus.cc:482] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.482321 18290 raft_consensus.cc:3037] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.483491 18290 raft_consensus.cc:504] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.483808 18290 leader_election.cc:288] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:05.485594 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c13af71c56c49c89f4ba43be9956a50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:05.485718 9535 raft_consensus.cc:3037] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.486793 9535 raft_consensus.cc:2454] T 3c13af71c56c49c89f4ba43be9956a50 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:05.487449 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c13af71c56c49c89f4ba43be9956a50" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:05.487557 9401 raft_consensus.cc:3037] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.488560 9401 raft_consensus.cc:2454] T 3c13af71c56c49c89f4ba43be9956a50 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:05.489117 9666 raft_consensus.cc:1261] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.489265 6815 leader_election.cc:302] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:05.489586 17364 consensus_queue.cc:1035] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.490293 18290 raft_consensus.cc:2781] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.490680 7138 raft_consensus.cc:1261] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.490856 18290 raft_consensus.cc:686] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:05.491122 17364 consensus_queue.cc:1035] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.491254 18290 consensus_queue.cc:237] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:05.493780 6737 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet 1cdbbef7b1374601a6b0865ffb76385e (table rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4]). Expected version 1 got 0
I20240501 07:59:05.494231 18129 mvcc.cc:205] Tried to move back new op lower bound from 7022798215118098432 to 7022798214619848704. Current Snapshot: MvccSnapshot[applied={T|T < 7022798215118098432}]
I20240501 07:59:05.494412 18340 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:05.494735 18340 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:05.495206 18339 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:05.495704 18339 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:05.499521 6730 catalog_manager.cc:5484] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:05.500725 18342 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:05.501026 18342 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:05.503073 6732 catalog_manager.cc:5576] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107) does not have the latest schema for tablet 1cdbbef7b1374601a6b0865ffb76385e (table rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4]). Expected version 1 got 0
I20240501 07:59:05.503398 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4] alter complete (version 1)
I20240501 07:59:05.522117 6857 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:05.532460 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:05.534157 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.560211 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:05.561600 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.599145 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:05.604075 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:05.609699 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:05.611179 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:05.618288 18339 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:05.618626 18339 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:05.618870 18340 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:05.619166 18340 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:05.624845 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=854d68f6709d4d8cb92f2658617bf8f4] alter complete (version 2)
I20240501 07:59:05.630100 18342 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:05.630436 18342 tablet.cc:1620] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:05.652686 16671 raft_consensus.cc:482] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.652819 16671 raft_consensus.cc:504] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.653172 16671 leader_election.cc:288] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:05.653472 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:05.653609 7270 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:05.653828 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:05.653918 9469 leader_election.cc:302] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:05.653959 9401 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:05.654083 16671 raft_consensus.cc:2781] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.654143 16671 raft_consensus.cc:482] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.654183 16671 raft_consensus.cc:3037] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.655177 16671 raft_consensus.cc:504] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.655541 16671 leader_election.cc:288] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:05.655930 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:05.656050 9401 raft_consensus.cc:3037] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.657178 9401 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:05.657167 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:05.657291 7270 raft_consensus.cc:3037] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.658159 9470 leader_election.cc:302] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:05.658308 16671 raft_consensus.cc:2781] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.658378 16671 raft_consensus.cc:686] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:05.658358 7270 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:05.658510 16671 consensus_queue.cc:237] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:05.661681 6737 catalog_manager.cc:5484] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:05.680209 9666 consensus_queue.cc:237] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:05.681412 7010 raft_consensus.cc:1261] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:05.681522 9535 raft_consensus.cc:1261] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:05.682260 15946 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:05.682483 17788 consensus_queue.cc:1035] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:05.685076 15946 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:05.685566 9535 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:05.687178 7010 raft_consensus.cc:2932] T d38445516d1e49d88ef9707246d70115 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:05.689074 6732 catalog_manager.cc:5484] T d38445516d1e49d88ef9707246d70115 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:05.692561 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet d38445516d1e49d88ef9707246d70115 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:05.692888 18380 tablet_replica.cc:330] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:05.693123 18380 raft_consensus.cc:2229] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.693264 18380 raft_consensus.cc:2258] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.693733 18380 ts_tablet_manager.cc:1875] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:05.696130 18380 ts_tablet_manager.cc:1888] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:05.696215 18380 log.cc:1196] T d38445516d1e49d88ef9707246d70115 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d38445516d1e49d88ef9707246d70115
I20240501 07:59:05.696842 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet d38445516d1e49d88ef9707246d70115 (table rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]) successfully deleted
I20240501 07:59:05.730742 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:05.737298 18339 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:05.738528 18340 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:05.738830 18342 tablet.cc:1722] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:05.744451 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=854d68f6709d4d8cb92f2658617bf8f4] alter complete (version 3)
I20240501 07:59:05.837800 17277 consensus_queue.cc:1035] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.846979 6857 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:05.847379 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.851289 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:05.855432 9250 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:05.857754 6729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:05.857895 6729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:05.864405 6729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 1cdbbef7b1374601a6b0865ffb76385e
I20240501 07:59:05.864969 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:05.865442 9646 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:05.866534 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdbbef7b1374601a6b0865ffb76385e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:05 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:05.869235 9646 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:05.872045 9534 raft_consensus.cc:3037] T 767acae4338b4b0dba53109bf86c13fc P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.874280 17815 consensus_queue.cc:1035] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.881669 17229 consensus_queue.cc:1035] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.884020 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:05.895390 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:05.897994 18432 tablet_replica.cc:330] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:05.898221 18432 raft_consensus.cc:2229] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.898553 18432 raft_consensus.cc:2258] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.899690 18290 consensus_queue.cc:1035] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.905999 18433 tablet_replica.cc:330] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:05.906271 18433 raft_consensus.cc:2229] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:05.906520 18433 raft_consensus.cc:2258] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.908695 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:05.918124 18431 tablet_replica.cc:330] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:05.918365 18431 raft_consensus.cc:2229] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.918519 18431 raft_consensus.cc:2258] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.918820 18433 ts_tablet_manager.cc:1875] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.919732 18431 ts_tablet_manager.cc:1875] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.921564 18433 ts_tablet_manager.cc:1888] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.921696 18433 log.cc:1196] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1cdbbef7b1374601a6b0865ffb76385e
I20240501 07:59:05.922223 18433 ts_tablet_manager.cc:1909] T 1cdbbef7b1374601a6b0865ffb76385e P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:05.922992 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 1cdbbef7b1374601a6b0865ffb76385e (table rebalancer_dynamic_table_new_name [id=854d68f6709d4d8cb92f2658617bf8f4]) successfully deleted
I20240501 07:59:05.931493 18432 ts_tablet_manager.cc:1875] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:05.932757 18431 ts_tablet_manager.cc:1888] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.932922 18431 log.cc:1196] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1cdbbef7b1374601a6b0865ffb76385e
I20240501 07:59:05.933281 18431 ts_tablet_manager.cc:1909] T 1cdbbef7b1374601a6b0865ffb76385e P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:59:05.933688 18432 ts_tablet_manager.cc:1888] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:05.934013 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 1cdbbef7b1374601a6b0865ffb76385e (table rebalancer_dynamic_table_new_name [id=854d68f6709d4d8cb92f2658617bf8f4]) successfully deleted
I20240501 07:59:05.934168 18432 log.cc:1196] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1cdbbef7b1374601a6b0865ffb76385e
I20240501 07:59:05.934471 18432 ts_tablet_manager.cc:1909] T 1cdbbef7b1374601a6b0865ffb76385e P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:59:05.937234 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 1cdbbef7b1374601a6b0865ffb76385e (table rebalancer_dynamic_table_new_name [id=854d68f6709d4d8cb92f2658617bf8f4]) successfully deleted
I20240501 07:59:05.959101 6876 consensus_queue.cc:237] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:05.960292 9666 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:05.960319 7010 raft_consensus.cc:1261] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:05.960649 17229 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:05.960826 18290 consensus_queue.cc:1035] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:05.962468 18458 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:05.962847 9666 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:05.965409 7010 raft_consensus.cc:2932] T 28f7eaacb7694552a8fa2f98a5a0c868 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:05.965962 6732 catalog_manager.cc:5484] T 28f7eaacb7694552a8fa2f98a5a0c868 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:05.968717 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f7eaacb7694552a8fa2f98a5a0c868 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:05.969049 18431 tablet_replica.cc:330] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:05.969218 18431 raft_consensus.cc:2229] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:05.969400 18431 raft_consensus.cc:2258] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:05.970890 18431 ts_tablet_manager.cc:1875] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:05.970821 6730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:05.973229 18431 ts_tablet_manager.cc:1888] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:05.974128 18431 log.cc:1196] T 28f7eaacb7694552a8fa2f98a5a0c868 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/28f7eaacb7694552a8fa2f98a5a0c868
I20240501 07:59:05.974697 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 28f7eaacb7694552a8fa2f98a5a0c868 (table rebalance_test_table_13 [id=0d147294927c496ab944eaa27b4eb7a9]) successfully deleted
I20240501 07:59:05.977169 9382 tablet_service.cc:1467] Processing CreateTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]), partition=
I20240501 07:59:05.977222 6989 tablet_service.cc:1467] Processing CreateTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]), partition=
I20240501 07:59:05.977288 9250 tablet_service.cc:1467] Processing CreateTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]), partition=
I20240501 07:59:05.977442 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93caef2b3c3406ba3d31b8bd75b3cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.977473 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93caef2b3c3406ba3d31b8bd75b3cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.977502 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93caef2b3c3406ba3d31b8bd75b3cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.979857 18464 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:05.980105 18465 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:05.980999 18465 tablet_bootstrap.cc:654] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.981498 18466 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:05.981652 18464 tablet_bootstrap.cc:654] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.982478 18466 tablet_bootstrap.cc:654] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.983566 18465 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:05.983705 18465 ts_tablet_manager.cc:1367] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:05.984676 18465 raft_consensus.cc:348] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.984833 18464 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:05.984946 18464 ts_tablet_manager.cc:1367] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:05.985587 18464 raft_consensus.cc:348] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.985747 18465 raft_consensus.cc:374] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.985749 18464 raft_consensus.cc:374] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.985803 18464 raft_consensus.cc:729] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.985898 18465 raft_consensus.cc:729] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.985965 18464 consensus_queue.cc:260] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.986105 18465 consensus_queue.cc:260] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.986410 18465 ts_tablet_manager.cc:1398] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:05.987038 18464 ts_tablet_manager.cc:1398] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.989284 18466 tablet_bootstrap.cc:492] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:05.990303 18466 ts_tablet_manager.cc:1367] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:05.990751 18466 raft_consensus.cc:348] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.990903 18466 raft_consensus.cc:374] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.990974 18466 raft_consensus.cc:729] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.991179 18466 consensus_queue.cc:260] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:05.991545 18466 ts_tablet_manager.cc:1398] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.091500 16671 consensus_queue.cc:1035] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.094666 18302 consensus_queue.cc:1035] T 84e079a65e864d8f8fc84761766a7420 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.173365 17815 raft_consensus.cc:482] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.173514 17815 raft_consensus.cc:504] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.173935 17815 leader_election.cc:288] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:06.174371 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93caef2b3c3406ba3d31b8bd75b3cc8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:06.174403 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93caef2b3c3406ba3d31b8bd75b3cc8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:06.174518 9401 raft_consensus.cc:2454] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:06.174538 9271 raft_consensus.cc:2454] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 0.
I20240501 07:59:06.174794 6945 leader_election.cc:302] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:06.174954 17815 raft_consensus.cc:2781] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.175040 17815 raft_consensus.cc:482] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.175094 17815 raft_consensus.cc:3037] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.176129 17815 raft_consensus.cc:504] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.176513 17815 leader_election.cc:288] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:06.176887 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93caef2b3c3406ba3d31b8bd75b3cc8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:06.177011 9271 raft_consensus.cc:3037] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.177199 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93caef2b3c3406ba3d31b8bd75b3cc8" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:06.177313 9401 raft_consensus.cc:3037] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.178071 9271 raft_consensus.cc:2454] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:06.178653 6945 leader_election.cc:302] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:06.178822 9401 raft_consensus.cc:2454] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:06.178865 17815 raft_consensus.cc:2781] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.178977 17815 raft_consensus.cc:686] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:06.179364 17815 consensus_queue.cc:237] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.180104 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:06.181890 6736 catalog_manager.cc:5484] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:06.186815 6857 tablet_service.cc:1467] Processing CreateTablet for tablet e6bc6feadf004640b29f30232a00e289 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.187152 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bc6feadf004640b29f30232a00e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.187579 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 25f4254fc0c148cebeb917e03b91aa31 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.187904 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f4254fc0c148cebeb917e03b91aa31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.189224 18510 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:06.189461 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 25f4254fc0c148cebeb917e03b91aa31 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.189743 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f4254fc0c148cebeb917e03b91aa31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.189893 18511 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:06.190207 9382 tablet_service.cc:1467] Processing CreateTablet for tablet e6bc6feadf004640b29f30232a00e289 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.190502 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bc6feadf004640b29f30232a00e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.190802 18510 tablet_bootstrap.cc:654] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.191352 9646 tablet_service.cc:1467] Processing CreateTablet for tablet e6bc6feadf004640b29f30232a00e289 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.191413 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 25f4254fc0c148cebeb917e03b91aa31 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.191612 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bc6feadf004640b29f30232a00e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.191677 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f4254fc0c148cebeb917e03b91aa31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.192471 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 7740c9898dae4a01bb3b567eee55f4b4 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.192692 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7740c9898dae4a01bb3b567eee55f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.192984 18464 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:06.193830 18466 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:06.193946 18464 tablet_bootstrap.cc:654] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.193989 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 7740c9898dae4a01bb3b567eee55f4b4 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.194206 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7740c9898dae4a01bb3b567eee55f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.194658 18466 tablet_bootstrap.cc:654] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.194659 18465 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:06.195617 18465 tablet_bootstrap.cc:654] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.195798 18511 tablet_bootstrap.cc:654] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.196385 18466 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:06.196478 18466 ts_tablet_manager.cc:1367] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:06.196985 18466 raft_consensus.cc:348] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.197086 18466 raft_consensus.cc:374] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.197099 18512 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:06.197117 18466 raft_consensus.cc:729] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.197261 18466 consensus_queue.cc:260] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.197563 18466 ts_tablet_manager.cc:1398] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.198177 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 7740c9898dae4a01bb3b567eee55f4b4 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=03f4394afc054c25a255ae527180e619]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:06.198410 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7740c9898dae4a01bb3b567eee55f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.199640 18465 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:06.199746 18465 ts_tablet_manager.cc:1367] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:06.200120 18465 raft_consensus.cc:348] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.200207 18465 raft_consensus.cc:374] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.200241 18465 raft_consensus.cc:729] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.200353 18465 consensus_queue.cc:260] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.200619 18465 ts_tablet_manager.cc:1398] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.200789 18465 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:06.201493 18464 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:06.201575 18464 ts_tablet_manager.cc:1367] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:06.201642 18465 tablet_bootstrap.cc:654] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.202040 18464 raft_consensus.cc:348] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.202143 18464 raft_consensus.cc:374] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.202194 18464 raft_consensus.cc:729] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.202311 18464 consensus_queue.cc:260] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.202590 18465 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:06.202631 18464 ts_tablet_manager.cc:1398] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.202661 18465 ts_tablet_manager.cc:1367] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:06.204282 18465 raft_consensus.cc:348] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.204375 18465 raft_consensus.cc:374] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.204397 18465 raft_consensus.cc:729] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.204495 18465 consensus_queue.cc:260] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.204743 18465 ts_tablet_manager.cc:1398] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:06.207073 18510 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:06.207168 18510 ts_tablet_manager.cc:1367] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20240501 07:59:06.207625 18510 raft_consensus.cc:348] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.207720 18510 raft_consensus.cc:374] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.207741 18510 raft_consensus.cc:729] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.207861 18510 consensus_queue.cc:260] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.208132 18510 ts_tablet_manager.cc:1398] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.215025 17229 raft_consensus.cc:482] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.215126 17229 raft_consensus.cc:504] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.215514 17229 leader_election.cc:288] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:06.215801 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:06.215921 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:06.215965 9401 raft_consensus.cc:2454] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
W20240501 07:59:06.216243 6814 leader_election.cc:341] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:06.218369 6814 leader_election.cc:302] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters: 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:06.218554 17229 raft_consensus.cc:2781] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.218616 17229 raft_consensus.cc:482] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.218668 17229 raft_consensus.cc:3037] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.219877 17229 raft_consensus.cc:504] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.220248 17229 leader_election.cc:288] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:06.220649 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:06.221349 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
W20240501 07:59:06.221417 6814 leader_election.cc:341] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:06.221475 9401 raft_consensus.cc:3037] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.222426 17449 raft_consensus.cc:482] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.222467 9401 raft_consensus.cc:2454] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:06.222522 17449 raft_consensus.cc:504] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.222961 17449 leader_election.cc:288] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:06.223390 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f4254fc0c148cebeb917e03b91aa31" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
W20240501 07:59:06.223790 9205 leader_election.cc:341] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:06.224895 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f4254fc0c148cebeb917e03b91aa31" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:06.225029 7010 raft_consensus.cc:2454] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:06.225644 9207 leader_election.cc:302] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters: 15ab96f6477b4291932409b80af07213
I20240501 07:59:06.225972 17449 raft_consensus.cc:2781] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.225972 6814 leader_election.cc:302] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters: 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:06.226052 17449 raft_consensus.cc:482] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.226106 17449 raft_consensus.cc:3037] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.226413 17229 raft_consensus.cc:2781] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.226562 17229 raft_consensus.cc:686] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:06.226698 17229 consensus_queue.cc:237] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.227253 17449 raft_consensus.cc:504] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.227640 17449 leader_election.cc:288] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:06.228120 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f4254fc0c148cebeb917e03b91aa31" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
W20240501 07:59:06.228435 9205 leader_election.cc:341] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:06.229307 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f4254fc0c148cebeb917e03b91aa31" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:06.229414 7010 raft_consensus.cc:3037] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.230437 18511 tablet_bootstrap.cc:492] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:06.230432 7010 raft_consensus.cc:2454] T 25f4254fc0c148cebeb917e03b91aa31 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:06.230528 18511 ts_tablet_manager.cc:1367] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.002s
I20240501 07:59:06.232717 9207 leader_election.cc:302] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters: 15ab96f6477b4291932409b80af07213
I20240501 07:59:06.232883 6735 catalog_manager.cc:5484] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:06.234182 17449 raft_consensus.cc:2781] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.234253 17449 raft_consensus.cc:686] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:06.234728 17449 consensus_queue.cc:237] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.237720 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:06.241060 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.241603 6730 catalog_manager.cc:5484] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:06.242462 18511 raft_consensus.cc:348] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.242587 18511 raft_consensus.cc:374] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.242621 18511 raft_consensus.cc:729] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.242738 18511 consensus_queue.cc:260] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.243151 18511 ts_tablet_manager.cc:1398] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 07:59:06.243335 18511 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:06.244796 7251 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:06.245198 18512 tablet_bootstrap.cc:654] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.252498 18511 tablet_bootstrap.cc:654] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.254586 18512 tablet_bootstrap.cc:492] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:06.254741 18512 ts_tablet_manager.cc:1367] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.000s
I20240501 07:59:06.255522 18512 raft_consensus.cc:348] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.255715 18512 raft_consensus.cc:374] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.255784 18512 raft_consensus.cc:729] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.255959 18512 consensus_queue.cc:260] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.256381 18512 ts_tablet_manager.cc:1398] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:06.257095 18512 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:06.257992 18512 tablet_bootstrap.cc:654] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.259104 18511 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:06.259233 18511 ts_tablet_manager.cc:1367] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 07:59:06.259713 18511 raft_consensus.cc:348] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.259877 18511 raft_consensus.cc:374] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.259946 18511 raft_consensus.cc:729] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.260133 18511 consensus_queue.cc:260] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.260531 18511 ts_tablet_manager.cc:1398] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.267192 9250 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.267940 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:06.269762 18512 tablet_bootstrap.cc:492] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:06.269923 18512 ts_tablet_manager.cc:1367] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:06.270527 18512 raft_consensus.cc:348] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.270750 18512 raft_consensus.cc:374] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.270833 18512 raft_consensus.cc:729] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.270901 6990 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:06.271049 18512 consensus_queue.cc:260] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.271523 18512 ts_tablet_manager.cc:1398] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.283210 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.300566 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:06.319514 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:06.323642 9271 raft_consensus.cc:1261] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.323660 9403 raft_consensus.cc:1261] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.324273 17277 consensus_queue.cc:1035] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.324540 17277 consensus_queue.cc:1035] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.327438 6730 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]). Expected version 1 got 0
I20240501 07:59:06.327952 6736 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]). Expected version 1 got 0
I20240501 07:59:06.333050 18541 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:06.333631 18541 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:06.336457 6736 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4]). Expected version 1 got 0
I20240501 07:59:06.339131 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4] alter complete (version 1)
I20240501 07:59:06.345503 18543 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:06.345984 18543 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:06.361872 18542 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:06.362243 18542 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:06.378598 9666 consensus_queue.cc:237] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:06.380198 7010 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:06.380182 9534 raft_consensus.cc:1261] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:06.380705 18375 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:06.380915 18560 consensus_queue.cc:1035] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:06.383208 18560 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:06.383749 7010 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:06.387470 6730 catalog_manager.cc:5484] T 4f3e4cd27e4545baa255a01023cf1d68 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 4 to 5, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:06.389930 9535 raft_consensus.cc:2932] T 4f3e4cd27e4545baa255a01023cf1d68 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:06.391067 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3e4cd27e4545baa255a01023cf1d68 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:06.391430 18563 tablet_replica.cc:330] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:06.391657 18563 raft_consensus.cc:2229] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:06.392027 18563 raft_consensus.cc:2258] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:06.392431 18563 ts_tablet_manager.cc:1875] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:06.394949 18563 ts_tablet_manager.cc:1888] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:06.395117 18563 log.cc:1196] T 4f3e4cd27e4545baa255a01023cf1d68 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4f3e4cd27e4545baa255a01023cf1d68
I20240501 07:59:06.395771 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 4f3e4cd27e4545baa255a01023cf1d68 (table rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]) successfully deleted
I20240501 07:59:06.440917 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:06.447625 18541 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:06.447865 18543 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:06.447902 18542 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:06.447985 18541 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: MemRowSet was empty: no flush needed.
I20240501 07:59:06.448146 18542 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:06.448146 18543 tablet.cc:1620] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:06.455209 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=8a964aad2a064950926c7106dd1291c4] alter complete (version 2)
I20240501 07:59:06.558249 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:06.564261 18541 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:06.565200 18543 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:06.572038 18542 tablet.cc:1722] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:06.575125 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=8a964aad2a064950926c7106dd1291c4] alter complete (version 3)
I20240501 07:59:06.631999 18532 consensus_queue.cc:1035] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.634699 7271 raft_consensus.cc:3037] T 25f4254fc0c148cebeb917e03b91aa31 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.636359 18532 consensus_queue.cc:1035] T 25f4254fc0c148cebeb917e03b91aa31 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.647886 18355 raft_consensus.cc:482] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.648015 18355 raft_consensus.cc:504] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.648417 18355 leader_election.cc:288] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:06.648779 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7740c9898dae4a01bb3b567eee55f4b4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:06.648913 7010 raft_consensus.cc:2454] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:06.649127 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7740c9898dae4a01bb3b567eee55f4b4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:06.649178 7208 leader_election.cc:302] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:06.649250 9665 raft_consensus.cc:2454] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:06.649338 18355 raft_consensus.cc:2781] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.649389 18355 raft_consensus.cc:482] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.649432 18355 raft_consensus.cc:3037] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.650383 18355 raft_consensus.cc:504] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.650693 18355 leader_election.cc:288] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:06.651042 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7740c9898dae4a01bb3b567eee55f4b4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:06.651099 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7740c9898dae4a01bb3b567eee55f4b4" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:06.651160 9665 raft_consensus.cc:3037] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.651206 7010 raft_consensus.cc:3037] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.652165 7010 raft_consensus.cc:2454] T 7740c9898dae4a01bb3b567eee55f4b4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:06.652165 9665 raft_consensus.cc:2454] T 7740c9898dae4a01bb3b567eee55f4b4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:06.652460 7207 leader_election.cc:302] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:06.652638 18355 raft_consensus.cc:2781] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.652706 18355 raft_consensus.cc:686] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:06.652829 18355 consensus_queue.cc:237] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:06.654990 6728 catalog_manager.cc:5484] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:06.658310 18564 raft_consensus.cc:482] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.658393 18564 raft_consensus.cc:504] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:06.658746 18564 leader_election.cc:288] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:06.659096 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:06.659185 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bc6feadf004640b29f30232a00e289" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:06.659351 9401 raft_consensus.cc:2379] T e6bc6feadf004640b29f30232a00e289 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fa3a88c25964445975bb47e0feed460 in current term 1: Already voted for candidate 1a9029bee6a144938f68b363e4cca162 in this term.
I20240501 07:59:06.659888 9601 leader_election.cc:302] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6
I20240501 07:59:06.660272 18565 raft_consensus.cc:3037] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.661206 18565 raft_consensus.cc:2726] T e6bc6feadf004640b29f30232a00e289 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:06.676618 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.678725 6728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:06.678879 6728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:06.681696 6728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet a93caef2b3c3406ba3d31b8bd75b3cc8
I20240501 07:59:06.682158 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:06 UTC) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:06.683064 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:06 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:06.683183 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet a93caef2b3c3406ba3d31b8bd75b3cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:06 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:06.683398 18563 tablet_replica.cc:330] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:06.683542 18563 raft_consensus.cc:2229] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:06.683686 18563 raft_consensus.cc:2258] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:06.683863 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:06.684693 9250 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.684814 18563 ts_tablet_manager.cc:1875] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:06.684932 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:06.686280 18624 tablet_replica.cc:330] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:06.686465 18624 raft_consensus.cc:2229] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:06.686604 18624 raft_consensus.cc:2258] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:06.687261 18624 ts_tablet_manager.cc:1875] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:06.687932 17229 consensus_queue.cc:1035] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.690287 18563 ts_tablet_manager.cc:1888] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:06.690407 18563 log.cc:1196] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a93caef2b3c3406ba3d31b8bd75b3cc8
I20240501 07:59:06.690742 18563 ts_tablet_manager.cc:1909] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:06.695822 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table_new_name [id=8a964aad2a064950926c7106dd1291c4]) successfully deleted
I20240501 07:59:06.695933 18458 consensus_queue.cc:1035] T e6bc6feadf004640b29f30232a00e289 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.697111 18623 tablet_replica.cc:330] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:06.697309 18623 raft_consensus.cc:2229] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:06.709996 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:06.712579 7121 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.713234 6990 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:06.714627 18623 raft_consensus.cc:2258] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:06.715399 18623 ts_tablet_manager.cc:1875] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:06.722991 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:06.744939 18624 ts_tablet_manager.cc:1888] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:06.745131 18624 log.cc:1196] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a93caef2b3c3406ba3d31b8bd75b3cc8
I20240501 07:59:06.745527 18624 ts_tablet_manager.cc:1909] T a93caef2b3c3406ba3d31b8bd75b3cc8 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:06.747339 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table_new_name [id=8a964aad2a064950926c7106dd1291c4]) successfully deleted
I20240501 07:59:06.748260 18623 ts_tablet_manager.cc:1888] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:06.748348 18623 log.cc:1196] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a93caef2b3c3406ba3d31b8bd75b3cc8
I20240501 07:59:06.748673 18623 ts_tablet_manager.cc:1909] T a93caef2b3c3406ba3d31b8bd75b3cc8 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting consensus metadata
I20240501 07:59:06.755136 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet a93caef2b3c3406ba3d31b8bd75b3cc8 (table rebalancer_dynamic_table_new_name [id=8a964aad2a064950926c7106dd1291c4]) successfully deleted
I20240501 07:59:06.783923 6728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:06.789938 9382 tablet_service.cc:1467] Processing CreateTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94]), partition=
I20240501 07:59:06.789990 9250 tablet_service.cc:1467] Processing CreateTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94]), partition=
I20240501 07:59:06.790215 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca65d2f26fa94b01aa9d126079a893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.790215 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca65d2f26fa94b01aa9d126079a893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.792675 7121 tablet_service.cc:1467] Processing CreateTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94]), partition=
I20240501 07:59:06.792944 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca65d2f26fa94b01aa9d126079a893e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:06.794432 18639 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:06.796845 18639 tablet_bootstrap.cc:654] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.798717 18639 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:06.798799 18639 ts_tablet_manager.cc:1367] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:06.799568 18639 raft_consensus.cc:348] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.799693 18639 raft_consensus.cc:374] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.799723 18641 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:06.799726 18639 raft_consensus.cc:729] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.799845 18639 consensus_queue.cc:260] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.800135 18639 ts_tablet_manager.cc:1398] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.801448 18640 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:06.800952 18641 tablet_bootstrap.cc:654] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.804630 18641 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:06.804984 18641 ts_tablet_manager.cc:1367] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:06.805369 18640 tablet_bootstrap.cc:654] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:06.807076 18641 raft_consensus.cc:348] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.807641 9534 consensus_queue.cc:237] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:06.808399 18641 raft_consensus.cc:374] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.808753 18641 raft_consensus.cc:729] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.808851 6877 raft_consensus.cc:1261] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:06.808867 9665 raft_consensus.cc:1261] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:06.809159 18641 consensus_queue.cc:260] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.809556 18641 ts_tablet_manager.cc:1398] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:06.809564 18302 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:06.809913 18302 consensus_queue.cc:1035] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:06.813644 16671 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:06.815603 18640 tablet_bootstrap.cc:492] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:06.815712 18640 ts_tablet_manager.cc:1367] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:06.816175 18640 raft_consensus.cc:348] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.816318 18640 raft_consensus.cc:374] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:06.816356 18640 raft_consensus.cc:729] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:06.816468 18640 consensus_queue.cc:260] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.816726 18640 ts_tablet_manager.cc:1398] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:06.817997 9665 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:06.820683 6878 raft_consensus.cc:2932] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:06.822654 6737 catalog_manager.cc:5484] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:06.825879 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3d606ace12402a86ab5ce85a98cd44 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:06.826968 18624 tablet_replica.cc:330] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:06.827154 18624 raft_consensus.cc:2229] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:06.827284 18624 raft_consensus.cc:2258] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:06.827723 18624 ts_tablet_manager.cc:1875] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:06.829984 18624 ts_tablet_manager.cc:1888] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:06.830051 18624 log.cc:1196] T 8c3d606ace12402a86ab5ce85a98cd44 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8c3d606ace12402a86ab5ce85a98cd44
I20240501 07:59:06.831205 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 8c3d606ace12402a86ab5ce85a98cd44 (table rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]) successfully deleted
I20240501 07:59:06.956779 17598 raft_consensus.cc:482] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.957044 17598 raft_consensus.cc:504] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.957422 17598 leader_election.cc:288] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:06.957823 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca65d2f26fa94b01aa9d126079a893e0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:06.957856 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca65d2f26fa94b01aa9d126079a893e0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:06.957952 9403 raft_consensus.cc:2454] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:06.958012 9271 raft_consensus.cc:2454] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:06.958299 7076 leader_election.cc:302] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:06.959725 17598 raft_consensus.cc:2781] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.959785 17598 raft_consensus.cc:482] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.959813 17598 raft_consensus.cc:3037] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.960597 17598 raft_consensus.cc:504] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.960877 17598 leader_election.cc:288] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:06.961155 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca65d2f26fa94b01aa9d126079a893e0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:06.961263 9403 raft_consensus.cc:3037] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.961248 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca65d2f26fa94b01aa9d126079a893e0" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:06.961388 9271 raft_consensus.cc:3037] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.962152 9403 raft_consensus.cc:2454] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:06.962287 9271 raft_consensus.cc:2454] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:06.962424 7076 leader_election.cc:302] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:06.962612 17598 raft_consensus.cc:2781] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.962738 17598 raft_consensus.cc:686] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:06.962905 17598 consensus_queue.cc:237] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:06.965693 6737 catalog_manager.cc:5484] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:06.991825 6990 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:06.999035 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:06.999631 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:07.002521 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:07.003962 7121 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.007980 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:07.013020 9382 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:07.029418 9250 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:07.092339 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:07.094211 7010 consensus_queue.cc:237] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:07.095520 9403 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:07.095538 9271 raft_consensus.cc:1261] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:07.095921 18588 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:07.096586 17277 consensus_queue.cc:1035] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:07.097997 9403 raft_consensus.cc:1261] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.098577 17598 consensus_queue.cc:1035] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.099179 9271 raft_consensus.cc:1261] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.099511 17598 consensus_queue.cc:1035] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.101758 18588 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.102687 6737 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet ca65d2f26fa94b01aa9d126079a893e0 (table rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94]). Expected version 1 got 0
I20240501 07:59:07.103049 9266 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.104060 18734 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.104710 9403 raft_consensus.cc:2932] T 651beb63e03d40bca88f01ebed416754 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.105161 18734 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:07.105526 18733 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.105840 18733 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:07.105851 6736 catalog_manager.cc:5576] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153) does not have the latest schema for tablet ca65d2f26fa94b01aa9d126079a893e0 (table rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94]). Expected version 1 got 0
I20240501 07:59:07.108453 6737 catalog_manager.cc:5484] T 651beb63e03d40bca88f01ebed416754 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.113656 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 651beb63e03d40bca88f01ebed416754 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58254
I20240501 07:59:07.114028 18739 tablet_replica.cc:330] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:59:07.114229 18739 raft_consensus.cc:2229] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.114758 18739 raft_consensus.cc:2258] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.115343 18740 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.115671 18740 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.116084 18739 ts_tablet_manager.cc:1875] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:07.118075 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94] alter complete (version 1)
I20240501 07:59:07.119441 18739 ts_tablet_manager.cc:1888] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:07.119596 18739 log.cc:1196] T 651beb63e03d40bca88f01ebed416754 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/651beb63e03d40bca88f01ebed416754
I20240501 07:59:07.120409 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 651beb63e03d40bca88f01ebed416754 (table rebalance_test_table_9 [id=eec42e978d1745efa8c79678ef5d0123]) successfully deleted
I20240501 07:59:07.146631 18393 consensus_queue.cc:1035] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.150563 18355 consensus_queue.cc:1035] T 7740c9898dae4a01bb3b567eee55f4b4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.211891 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:07.220619 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 70edef5951b74998b393ca86586abf61 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.220958 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70edef5951b74998b393ca86586abf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.221880 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20898d4ec84c34ac4234fee64fdadd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.221969 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffa33e44b45e8bdb1a71544e28250 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.222131 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20898d4ec84c34ac4234fee64fdadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.222397 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffa33e44b45e8bdb1a71544e28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.223534 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20898d4ec84c34ac4234fee64fdadd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.223798 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20898d4ec84c34ac4234fee64fdadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.224542 18640 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:07.225634 18640 tablet_bootstrap.cc:654] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.226352 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffa33e44b45e8bdb1a71544e28250 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.226604 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffa33e44b45e8bdb1a71544e28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.229225 18639 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:07.230096 18639 tablet_bootstrap.cc:654] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.230525 18640 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:07.230608 18640 ts_tablet_manager.cc:1367] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:07.231747 18640 raft_consensus.cc:348] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.231870 18640 raft_consensus.cc:374] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.231899 18640 raft_consensus.cc:729] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.231938 9515 tablet_service.cc:1467] Processing CreateTablet for tablet 70edef5951b74998b393ca86586abf61 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.232005 18640 consensus_queue.cc:260] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.232182 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70edef5951b74998b393ca86586abf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.232319 18640 ts_tablet_manager.cc:1398] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:07.232489 18640 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:07.233595 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeffa33e44b45e8bdb1a71544e28250 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.233829 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeffa33e44b45e8bdb1a71544e28250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.234898 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:07.234973 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 70edef5951b74998b393ca86586abf61 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.235237 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70edef5951b74998b393ca86586abf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.236476 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 9a20898d4ec84c34ac4234fee64fdadd (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=e651687ad9654543b18a1f8d2d9445c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.236749 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a20898d4ec84c34ac4234fee64fdadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.238914 18641 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:07.239825 18641 tablet_bootstrap.cc:654] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.241318 18640 tablet_bootstrap.cc:654] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.249781 18639 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:07.249888 18639 ts_tablet_manager.cc:1367] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:59:07.250376 18639 raft_consensus.cc:348] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.250494 18639 raft_consensus.cc:374] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.250526 18639 raft_consensus.cc:729] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.250658 18639 consensus_queue.cc:260] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.250972 18639 ts_tablet_manager.cc:1398] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.254478 18733 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.254787 18733 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: MemRowSet was empty: no flush needed.
I20240501 07:59:07.255736 18801 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:07.255982 18797 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:07.256107 18804 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:07.256891 18734 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.257205 18734 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:07.258453 18804 tablet_bootstrap.cc:654] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.259649 18800 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:07.261531 18740 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.261828 18740 tablet.cc:1620] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.263613 6728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=15b555293ea642bab77680a04ac5ba94] alter complete (version 2)
I20240501 07:59:07.265044 18797 tablet_bootstrap.cc:654] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.265054 18640 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:07.265136 18640 ts_tablet_manager.cc:1367] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.001s
I20240501 07:59:07.265333 18801 tablet_bootstrap.cc:654] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.265627 18640 raft_consensus.cc:348] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.265755 18640 raft_consensus.cc:374] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.265789 18640 raft_consensus.cc:729] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.265899 18640 consensus_queue.cc:260] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.266183 18640 ts_tablet_manager.cc:1398] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:07.267673 18641 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:07.267769 18641 ts_tablet_manager.cc:1367] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 07:59:07.268242 18641 raft_consensus.cc:348] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.268348 18641 raft_consensus.cc:374] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.268380 18641 raft_consensus.cc:729] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.268489 18641 consensus_queue.cc:260] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.268806 18641 ts_tablet_manager.cc:1398] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.276904 18800 tablet_bootstrap.cc:654] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.278324 18800 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:07.278553 18800 ts_tablet_manager.cc:1367] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 07:59:07.279343 18801 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:07.279557 18801 ts_tablet_manager.cc:1367] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:07.280047 18801 raft_consensus.cc:348] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.280160 18801 raft_consensus.cc:374] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.280211 18801 raft_consensus.cc:729] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.280341 18801 consensus_queue.cc:260] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.280644 18801 ts_tablet_manager.cc:1398] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:07.283655 18800 raft_consensus.cc:348] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.283828 18800 raft_consensus.cc:374] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.284055 18800 raft_consensus.cc:729] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.284241 18800 consensus_queue.cc:260] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.286676 18804 tablet_bootstrap.cc:492] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:07.286774 18804 ts_tablet_manager.cc:1367] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240501 07:59:07.287287 18804 raft_consensus.cc:348] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.287387 18804 raft_consensus.cc:374] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.287420 18804 raft_consensus.cc:729] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.287551 18804 consensus_queue.cc:260] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.287873 18804 ts_tablet_manager.cc:1398] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.288384 17449 raft_consensus.cc:482] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.288481 17449 raft_consensus.cc:504] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.288704 18797 tablet_bootstrap.cc:492] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:07.288777 18797 ts_tablet_manager.cc:1367] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240501 07:59:07.288836 17449 leader_election.cc:288] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:07.289137 18797 raft_consensus.cc:348] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.289229 18797 raft_consensus.cc:374] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.289249 18797 raft_consensus.cc:729] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.289371 18797 consensus_queue.cc:260] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.289428 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:07.289469 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:07.289547 9401 raft_consensus.cc:2454] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 0.
I20240501 07:59:07.289645 18797 ts_tablet_manager.cc:1398] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:59:07.289819 9207 leader_election.cc:341] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:07.289930 9206 leader_election.cc:302] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:07.290190 17449 raft_consensus.cc:2781] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.290241 17449 raft_consensus.cc:482] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.290278 17449 raft_consensus.cc:3037] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.291532 17449 raft_consensus.cc:504] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.291847 17449 leader_election.cc:288] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:07.292686 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:07.292794 9401 raft_consensus.cc:3037] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.293650 18800 ts_tablet_manager.cc:1398] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 07:59:07.293843 9401 raft_consensus.cc:2454] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:07.294194 9206 leader_election.cc:302] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:07.294533 18532 raft_consensus.cc:2781] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.294602 18532 raft_consensus.cc:686] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:07.294710 18532 consensus_queue.cc:237] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.295498 18800 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:07.297812 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:07.298069 17891 raft_consensus.cc:482] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.298178 17891 raft_consensus.cc:504] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.298570 17891 leader_election.cc:288] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:07.298967 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20898d4ec84c34ac4234fee64fdadd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:07.299109 6877 raft_consensus.cc:2454] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:07.298867 6733 catalog_manager.cc:5484] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:07.299597 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20898d4ec84c34ac4234fee64fdadd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:07.299718 9403 raft_consensus.cc:2454] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:07.299764 18800 tablet_bootstrap.cc:654] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.300026 7076 leader_election.cc:302] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:07.300225 9207 leader_election.cc:341] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:07.300323 17598 raft_consensus.cc:2781] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.300370 17598 raft_consensus.cc:482] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.300407 17598 raft_consensus.cc:3037] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.301474 17598 raft_consensus.cc:504] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.301566 18800 tablet_bootstrap.cc:492] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:07.301648 18800 ts_tablet_manager.cc:1367] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 07:59:07.301797 17598 leader_election.cc:288] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:07.302122 18800 raft_consensus.cc:348] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.302232 18800 raft_consensus.cc:374] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.302266 18800 raft_consensus.cc:729] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.302393 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20898d4ec84c34ac4234fee64fdadd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:07.302390 18800 consensus_queue.cc:260] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.302500 6877 raft_consensus.cc:3037] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.302706 18800 ts_tablet_manager.cc:1398] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.303503 6877 raft_consensus.cc:2454] T 9a20898d4ec84c34ac4234fee64fdadd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:07.303787 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a20898d4ec84c34ac4234fee64fdadd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:07.303912 9403 raft_consensus.cc:3037] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.304819 9403 raft_consensus.cc:2454] T 9a20898d4ec84c34ac4234fee64fdadd P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:07.305644 7076 leader_election.cc:302] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:07.305928 17598 raft_consensus.cc:2781] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.306023 17598 raft_consensus.cc:686] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:07.306154 17598 consensus_queue.cc:237] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.314424 6733 catalog_manager.cc:5484] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:07.322018 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:07.325212 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:07.329663 9507 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:07.332489 18302 raft_consensus.cc:482] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.332607 18302 raft_consensus.cc:504] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.333240 18302 leader_election.cc:288] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:07.333884 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:07.333884 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeffa33e44b45e8bdb1a71544e28250" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:07.334045 9403 raft_consensus.cc:2379] T 6eeffa33e44b45e8bdb1a71544e28250 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac6c8bc1b74b483791e262abee7862cf in current term 1: Already voted for candidate 49adb9e99822494196ed22044bfed81f in this term.
I20240501 07:59:07.334642 9470 leader_election.cc:302] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf; no voters: 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6
I20240501 07:59:07.335471 18653 raft_consensus.cc:3037] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.336444 18653 raft_consensus.cc:2726] T 6eeffa33e44b45e8bdb1a71544e28250 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:07.341375 6990 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.341631 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.342602 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:07.348345 7121 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:07.351334 18302 raft_consensus.cc:482] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.351425 18302 raft_consensus.cc:504] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.351828 18302 leader_election.cc:288] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:07.352113 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:07.352252 7010 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:07.352555 9471 leader_election.cc:302] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:07.352708 18302 raft_consensus.cc:2781] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.352756 18302 raft_consensus.cc:482] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.352797 18302 raft_consensus.cc:3037] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.353844 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:07.353855 18302 raft_consensus.cc:504] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.353971 7271 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:07.354413 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:07.354535 7010 raft_consensus.cc:3037] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.355556 7010 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:07.356299 9471 leader_election.cc:302] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:07.356467 18653 raft_consensus.cc:2781] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.356544 18653 raft_consensus.cc:686] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:07.356688 18653 consensus_queue.cc:237] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.360169 18302 leader_election.cc:288] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:07.360558 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:07.360688 7271 raft_consensus.cc:3037] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.361685 7271 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:07.368613 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:07.371098 6733 catalog_manager.cc:5484] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:07.379194 18733 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.383105 18740 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.384907 18734 tablet.cc:1722] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.399524 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:07.398092 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=15b555293ea642bab77680a04ac5ba94] alter complete (version 3)
I20240501 07:59:07.457849 7140 consensus_queue.cc:237] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:07.459635 9665 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:07.459652 9270 raft_consensus.cc:1261] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:07.460361 17598 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:07.460566 17891 consensus_queue.cc:1035] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:07.462636 17891 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.463511 9665 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.466779 9270 raft_consensus.cc:2932] T 82c30f6292754d97a839700be93f8c07 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.467947 6728 catalog_manager.cc:5484] T 82c30f6292754d97a839700be93f8c07 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:07.472282 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c30f6292754d97a839700be93f8c07 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:07.472678 18849 tablet_replica.cc:330] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:07.472920 18849 raft_consensus.cc:2229] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.473361 18849 raft_consensus.cc:2258] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.473980 18849 ts_tablet_manager.cc:1875] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:07.476614 18849 ts_tablet_manager.cc:1888] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:07.476773 18849 log.cc:1196] T 82c30f6292754d97a839700be93f8c07 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/82c30f6292754d97a839700be93f8c07
I20240501 07:59:07.477556 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 82c30f6292754d97a839700be93f8c07 (table rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]) successfully deleted
I20240501 07:59:07.509568 6731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:07.509735 6731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:07.512023 6731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet ca65d2f26fa94b01aa9d126079a893e0
I20240501 07:59:07.512539 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:07.512634 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:07.512655 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet ca65d2f26fa94b01aa9d126079a893e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:07.517359 18866 tablet_replica.cc:330] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:07.517756 18866 raft_consensus.cc:2229] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:07.523236 18866 raft_consensus.cc:2258] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.523986 18866 ts_tablet_manager.cc:1875] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.526794 18868 tablet_replica.cc:330] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:07.526994 18868 raft_consensus.cc:2229] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.527844 18868 raft_consensus.cc:2258] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.528812 18866 ts_tablet_manager.cc:1888] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.530678 18868 ts_tablet_manager.cc:1875] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.531451 18866 log.cc:1196] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca65d2f26fa94b01aa9d126079a893e0
I20240501 07:59:07.531911 18866 ts_tablet_manager.cc:1909] T ca65d2f26fa94b01aa9d126079a893e0 P fc946354b5d04503a56122e59fc320be: Deleting consensus metadata
I20240501 07:59:07.534456 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet ca65d2f26fa94b01aa9d126079a893e0 (table rebalancer_dynamic_table_new_name [id=15b555293ea642bab77680a04ac5ba94]) successfully deleted
I20240501 07:59:07.538079 18868 ts_tablet_manager.cc:1888] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.538339 18868 log.cc:1196] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ca65d2f26fa94b01aa9d126079a893e0
I20240501 07:59:07.538604 18867 tablet_replica.cc:330] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:07.538933 18868 ts_tablet_manager.cc:1909] T ca65d2f26fa94b01aa9d126079a893e0 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:07.539130 18867 raft_consensus.cc:2229] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.539654 18867 raft_consensus.cc:2258] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.540278 18867 ts_tablet_manager.cc:1875] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.540714 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet ca65d2f26fa94b01aa9d126079a893e0 (table rebalancer_dynamic_table_new_name [id=15b555293ea642bab77680a04ac5ba94]) successfully deleted
I20240501 07:59:07.543260 18867 ts_tablet_manager.cc:1888] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.543383 18867 log.cc:1196] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ca65d2f26fa94b01aa9d126079a893e0
I20240501 07:59:07.543774 18867 ts_tablet_manager.cc:1909] T ca65d2f26fa94b01aa9d126079a893e0 P fa6964452854415a85ad72f8639768d6: Deleting consensus metadata
I20240501 07:59:07.544504 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet ca65d2f26fa94b01aa9d126079a893e0 (table rebalancer_dynamic_table_new_name [id=15b555293ea642bab77680a04ac5ba94]) successfully deleted
I20240501 07:59:07.615234 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:07.622680 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]), partition=
I20240501 07:59:07.622965 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ded6f45a3734c02a454b1317b6bfa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.625387 18639 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:07.625557 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]), partition=
I20240501 07:59:07.625814 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ded6f45a3734c02a454b1317b6bfa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.626327 18639 tablet_bootstrap.cc:654] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.627238 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]), partition=
I20240501 07:59:07.627473 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ded6f45a3734c02a454b1317b6bfa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.629314 18801 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:07.630167 18801 tablet_bootstrap.cc:654] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.633160 18901 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:07.633802 18639 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:07.633898 18639 ts_tablet_manager.cc:1367] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:07.634303 18901 tablet_bootstrap.cc:654] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.634402 18639 raft_consensus.cc:348] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.634517 18639 raft_consensus.cc:374] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.634553 18639 raft_consensus.cc:729] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.634691 18639 consensus_queue.cc:260] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.635015 18639 ts_tablet_manager.cc:1398] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.635560 18801 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:07.635639 18901 tablet_bootstrap.cc:492] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:07.635659 18801 ts_tablet_manager.cc:1367] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:07.635710 18901 ts_tablet_manager.cc:1367] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:07.636152 18901 raft_consensus.cc:348] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.637681 18901 raft_consensus.cc:374] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.637730 18901 raft_consensus.cc:729] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.636276 18801 raft_consensus.cc:348] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.638054 18901 consensus_queue.cc:260] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.638293 18801 raft_consensus.cc:374] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.638352 18801 raft_consensus.cc:729] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.638463 18801 consensus_queue.cc:260] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.638540 18901 ts_tablet_manager.cc:1398] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:07.643605 18801 ts_tablet_manager.cc:1398] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 07:59:07.669412 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:07.670456 7251 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:07.672207 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:07.682961 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.696465 9507 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:07.699698 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:07.716643 6990 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.730654 7121 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:07.756345 18564 raft_consensus.cc:482] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.756469 18564 raft_consensus.cc:504] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.756912 18564 leader_election.cc:288] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:07.757289 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded6f45a3734c02a454b1317b6bfa20" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:07.757511 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded6f45a3734c02a454b1317b6bfa20" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:07.757640 7271 raft_consensus.cc:2454] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:07.757848 9271 raft_consensus.cc:2454] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:07.758095 9601 leader_election.cc:302] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:07.758251 18564 raft_consensus.cc:2781] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.758306 18564 raft_consensus.cc:482] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.758340 18564 raft_consensus.cc:3037] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.759311 18564 raft_consensus.cc:504] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.759685 18564 leader_election.cc:288] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:07.760216 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded6f45a3734c02a454b1317b6bfa20" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:07.760360 9270 raft_consensus.cc:3037] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.761427 9270 raft_consensus.cc:2454] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:07.761659 18532 consensus_queue.cc:1035] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.762055 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded6f45a3734c02a454b1317b6bfa20" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:07.762178 7271 raft_consensus.cc:3037] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.762187 9601 leader_election.cc:302] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:07.763969 7271 raft_consensus.cc:2454] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:07.765707 18532 consensus_queue.cc:1035] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.766337 18564 raft_consensus.cc:2781] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.766423 18564 raft_consensus.cc:686] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:07.766558 18564 consensus_queue.cc:237] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:07.771607 6736 catalog_manager.cc:5484] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:07.800637 7270 consensus_queue.cc:237] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:07.801941 9666 raft_consensus.cc:1261] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:07.802354 9535 raft_consensus.cc:1261] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:07.803098 18355 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:07.803221 17891 consensus_queue.cc:1035] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.803442 18355 consensus_queue.cc:1035] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:07.805305 18393 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:07.807552 9666 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:07.810792 9535 raft_consensus.cc:2932] T 2ba12580263f4bd883ad7e003ebd6c51 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:07.811228 6732 catalog_manager.cc:5484] T 2ba12580263f4bd883ad7e003ebd6c51 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:07.816668 17598 consensus_queue.cc:1035] T 9a20898d4ec84c34ac4234fee64fdadd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.816788 6857 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba12580263f4bd883ad7e003ebd6c51 with delete_type TABLET_DATA_TOMBSTONED (TS 1a9029bee6a144938f68b363e4cca162 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54266
I20240501 07:59:07.817396 18943 tablet_replica.cc:330] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: stopping tablet replica
I20240501 07:59:07.817610 18943 raft_consensus.cc:2229] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.817977 18943 raft_consensus.cc:2258] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.820079 18943 ts_tablet_manager.cc:1875] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:07.824560 18943 ts_tablet_manager.cc:1888] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:07.824640 18943 log.cc:1196] T 2ba12580263f4bd883ad7e003ebd6c51 P 1a9029bee6a144938f68b363e4cca162: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ba12580263f4bd883ad7e003ebd6c51
I20240501 07:59:07.825227 6723 catalog_manager.cc:4830] TS 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): tablet 2ba12580263f4bd883ad7e003ebd6c51 (table rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]) successfully deleted
I20240501 07:59:07.830250 18302 consensus_queue.cc:1035] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.834177 18653 consensus_queue.cc:1035] T 70edef5951b74998b393ca86586abf61 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.854403 9535 consensus_queue.cc:237] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:07.855850 9666 raft_consensus.cc:1261] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.855983 9271 raft_consensus.cc:1261] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.856261 18302 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:07.856490 9471 consensus_peers.cc:485] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a4ad48cbfb947f4aec462093006f6c7. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:07.856487 16671 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:07.858345 18302 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:07.858810 9271 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:07.860849 9666 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:07.864980 6736 catalog_manager.cc:5484] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:07.883488 6731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:07.884001 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:07.888936 9535 consensus_queue.cc:237] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:07.890712 9271 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:07.891047 9470 consensus_peers.cc:485] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d30d4d6ddf3a40989b10cf5c10ef7082. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:07.891201 9401 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.891278 18302 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:07.892114 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebd607453c4cb6a0d4762dfddc1464 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.892117 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebd607453c4cb6a0d4762dfddc1464 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.892117 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebd607453c4cb6a0d4762dfddc1464 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.892388 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebd607453c4cb6a0d4762dfddc1464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.892388 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebd607453c4cb6a0d4762dfddc1464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.892393 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebd607453c4cb6a0d4762dfddc1464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.893728 6990 tablet_service.cc:1467] Processing CreateTablet for tablet eae3ef1d35a2437692eed3696ebd6649 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.893994 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae3ef1d35a2437692eed3696ebd6649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.895871 7121 tablet_service.cc:1467] Processing CreateTablet for tablet eae3ef1d35a2437692eed3696ebd6649 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.895938 7251 tablet_service.cc:1467] Processing CreateTablet for tablet eae3ef1d35a2437692eed3696ebd6649 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.896149 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae3ef1d35a2437692eed3696ebd6649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.896225 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae3ef1d35a2437692eed3696ebd6649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.896576 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.896844 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.897666 18994 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:07.898597 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.898833 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.899152 18801 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:07.899375 18994 tablet_bootstrap.cc:654] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.899727 18997 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:07.900108 18801 tablet_bootstrap.cc:654] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.900730 18997 tablet_bootstrap.cc:654] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.900774 18639 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:07.901733 18639 tablet_bootstrap.cc:654] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.901782 18992 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:07.902236 18997 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:07.902318 18997 ts_tablet_manager.cc:1367] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:07.902443 7270 raft_consensus.cc:1261] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.902776 18997 raft_consensus.cc:348] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.902894 18997 raft_consensus.cc:374] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.902837 18565 consensus_queue.cc:1035] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.902937 18997 raft_consensus.cc:729] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.902951 18993 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:07.903056 18302 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:07.903234 18997 consensus_queue.cc:260] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.903530 18992 tablet_bootstrap.cc:654] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.903707 18997 ts_tablet_manager.cc:1398] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.904134 9270 raft_consensus.cc:1261] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.905464 18565 consensus_queue.cc:1035] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.907050 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=0604e67552874c6b8eefde940c9aafd0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.907317 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd8c956cd084349af9d6a7c2b5fc7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.907449 6736 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]). Expected version 1 got 0
I20240501 07:59:07.911764 18990 mvcc.cc:205] Tried to move back new op lower bound from 7022798225004384256 to 7022798224453455872. Current Snapshot: MvccSnapshot[applied={T|T < 7022798225004384256}]
I20240501 07:59:07.912813 6731 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]). Expected version 1 got 0
I20240501 07:59:07.913946 18639 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:07.914042 18639 ts_tablet_manager.cc:1367] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:07.914386 18801 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:07.914484 18801 ts_tablet_manager.cc:1367] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240501 07:59:07.914494 18639 raft_consensus.cc:348] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.914606 18639 raft_consensus.cc:374] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.914638 18639 raft_consensus.cc:729] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.914748 18639 consensus_queue.cc:260] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.914934 18801 raft_consensus.cc:348] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.915067 18639 ts_tablet_manager.cc:1398] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.915112 18801 raft_consensus.cc:374] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.915158 18801 raft_consensus.cc:729] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.915278 18801 consensus_queue.cc:260] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.915571 18801 ts_tablet_manager.cc:1398] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.915737 18801 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:07.916059 18993 tablet_bootstrap.cc:654] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.916710 18801 tablet_bootstrap.cc:654] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.916754 19006 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.917035 19006 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.917395 19005 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.917677 19005 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:07.918639 18993 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:07.918720 18993 ts_tablet_manager.cc:1367] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:07.919363 6736 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]). Expected version 1 got 0
I20240501 07:59:07.920866 19007 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.921111 18801 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:07.921193 18801 ts_tablet_manager.cc:1367] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:07.921253 19007 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:59:07.921550 18993 raft_consensus.cc:348] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.921653 18801 raft_consensus.cc:348] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.921758 18801 raft_consensus.cc:374] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.921797 18801 raft_consensus.cc:729] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.921911 18801 consensus_queue.cc:260] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.922184 18801 ts_tablet_manager.cc:1398] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.922379 18992 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:07.922480 18992 ts_tablet_manager.cc:1367] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:59:07.922950 18993 raft_consensus.cc:374] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.923100 18993 raft_consensus.cc:729] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.923285 18993 consensus_queue.cc:260] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.923969 18994 tablet_bootstrap.cc:492] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:07.924062 18994 ts_tablet_manager.cc:1367] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20240501 07:59:07.924080 6736 catalog_manager.cc:5576] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309) does not have the latest schema for tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5]). Expected version 1 got 0
I20240501 07:59:07.924533 18993 ts_tablet_manager.cc:1398] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:59:07.924552 18994 raft_consensus.cc:348] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.924652 18994 raft_consensus.cc:374] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.924687 18994 raft_consensus.cc:729] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.924767 18993 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:07.924803 18994 consensus_queue.cc:260] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.925081 18994 ts_tablet_manager.cc:1398] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:07.925251 18994 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:07.926177 18992 raft_consensus.cc:348] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.926291 18992 raft_consensus.cc:374] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.926328 18992 raft_consensus.cc:729] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.926551 18992 consensus_queue.cc:260] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.926718 18994 tablet_bootstrap.cc:654] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.926843 18992 ts_tablet_manager.cc:1398] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:07.928582 18993 tablet_bootstrap.cc:654] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.930876 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5] alter complete (version 1)
I20240501 07:59:07.932601 18993 tablet_bootstrap.cc:492] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:07.932701 18993 ts_tablet_manager.cc:1367] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 07:59:07.933161 18993 raft_consensus.cc:348] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.933274 18993 raft_consensus.cc:374] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.933308 18993 raft_consensus.cc:729] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.933439 18993 consensus_queue.cc:260] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:07.933749 18993 ts_tablet_manager.cc:1398] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:07.934882 18302 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:07.936316 18994 tablet_bootstrap.cc:492] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:07.936730 9401 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:07.937302 9271 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:07.939281 6736 catalog_manager.cc:5484] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:07.936704 18994 ts_tablet_manager.cc:1367] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 07:59:07.943845 18994 raft_consensus.cc:348] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.944044 18994 raft_consensus.cc:374] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.944384 18994 raft_consensus.cc:729] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.944540 18994 consensus_queue.cc:260] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:07.945192 18994 ts_tablet_manager.cc:1398] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:07.962765 9401 consensus_queue.cc:237] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:07.964162 9666 raft_consensus.cc:1261] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.964262 6877 raft_consensus.cc:1261] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:07.964463 9337 consensus_peers.cc:485] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 010c12e2cc194cdb87ff9299a8e42a77. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:07.964860 19031 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:07.965121 17364 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:07.967154 17364 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:07.967768 9666 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:07.971874 6731 catalog_manager.cc:5484] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:07.974519 6877 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:07.990443 7270 consensus_queue.cc:237] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
W20240501 07:59:07.992192 7207 consensus_peers.cc:485] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e7f613e16c245de83fde8c277b8d350. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:07.992285 9666 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:07.992606 7138 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:07.993060 18938 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:07.993270 18937 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:07.995795 18937 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:07.996294 7138 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:07.997859 9666 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:08.000362 6731 catalog_manager.cc:5484] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 5 to 6, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:08.002082 17891 raft_consensus.cc:482] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.002171 17891 raft_consensus.cc:504] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.002530 17891 leader_election.cc:288] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:08.002866 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cebd607453c4cb6a0d4762dfddc1464" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:08.003002 6877 raft_consensus.cc:2454] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.003037 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cebd607453c4cb6a0d4762dfddc1464" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:08.003180 9401 raft_consensus.cc:2454] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.003485 7076 leader_election.cc:302] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.003810 17891 raft_consensus.cc:2781] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.003875 17891 raft_consensus.cc:482] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.003921 17891 raft_consensus.cc:3037] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.004961 17891 raft_consensus.cc:504] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.005301 17891 leader_election.cc:288] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:08.005573 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cebd607453c4cb6a0d4762dfddc1464" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:08.005703 6877 raft_consensus.cc:3037] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.005846 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cebd607453c4cb6a0d4762dfddc1464" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:08.005957 9401 raft_consensus.cc:3037] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.006673 6877 raft_consensus.cc:2454] T 1cebd607453c4cb6a0d4762dfddc1464 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.006902 9401 raft_consensus.cc:2454] T 1cebd607453c4cb6a0d4762dfddc1464 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.006978 7076 leader_election.cc:302] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.007165 17891 raft_consensus.cc:2781] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.007270 17891 raft_consensus.cc:686] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:08.007416 17891 consensus_queue.cc:237] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.010104 6732 catalog_manager.cc:5484] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:08.031131 7270 consensus_queue.cc:237] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.032637 7009 raft_consensus.cc:1261] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.032667 7138 raft_consensus.cc:1261] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.033071 18938 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:08.033267 7208 consensus_peers.cc:485] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9051ba90bad3430fa2e128c88ec7b6f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.033433 18355 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.035101 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:08.035353 18393 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.035856 7138 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.038465 7009 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.040501 6736 catalog_manager.cc:5484] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.043473 19005 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.043828 19005 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:08.044974 19006 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.045274 19006 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:08.046478 19007 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.046770 19007 tablet.cc:1620] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: MemRowSet was empty: no flush needed.
I20240501 07:59:08.051748 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=031f4d129c7a47029a63689d2855d2a5] alter complete (version 2)
I20240501 07:59:08.080085 9535 consensus_queue.cc:237] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.081554 9403 raft_consensus.cc:1261] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.081662 9270 raft_consensus.cc:1261] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:08.081862 9470 consensus_peers.cc:485] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81e4af3cf8f044e2be0f8e3195f8c084. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.082073 18302 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:08.082286 19013 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:08.084996 18302 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:08.086747 9403 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:08.087116 18355 raft_consensus.cc:482] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.087222 18355 raft_consensus.cc:504] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:08.087301 9270 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:08.087591 18355 leader_election.cc:288] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:08.087917 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae3ef1d35a2437692eed3696ebd6649" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:08.088044 7138 raft_consensus.cc:2454] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:08.088356 7206 leader_election.cc:302] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.088538 18355 raft_consensus.cc:2781] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.088608 18355 raft_consensus.cc:482] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.088654 18355 raft_consensus.cc:3037] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.088804 6732 catalog_manager.cc:5484] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:08.089620 18355 raft_consensus.cc:504] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:08.089682 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae3ef1d35a2437692eed3696ebd6649" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:08.089819 7009 raft_consensus.cc:2454] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:08.090000 18355 leader_election.cc:288] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:08.090730 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae3ef1d35a2437692eed3696ebd6649" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:08.090785 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae3ef1d35a2437692eed3696ebd6649" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:08.090842 7138 raft_consensus.cc:3037] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.090883 7009 raft_consensus.cc:3037] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.091738 7009 raft_consensus.cc:2454] T eae3ef1d35a2437692eed3696ebd6649 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:08.091866 7138 raft_consensus.cc:2454] T eae3ef1d35a2437692eed3696ebd6649 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:08.091991 7208 leader_election.cc:302] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:08.092216 18355 raft_consensus.cc:2781] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.092312 18355 raft_consensus.cc:686] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:08.092429 18355 consensus_queue.cc:237] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:08.095059 6736 catalog_manager.cc:5484] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:08.109077 18393 raft_consensus.cc:482] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.109203 18393 raft_consensus.cc:504] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.109634 18393 leader_election.cc:288] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:08.110641 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd8c956cd084349af9d6a7c2b5fc7a1" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:08.110776 9270 raft_consensus.cc:2454] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:08.110755 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd8c956cd084349af9d6a7c2b5fc7a1" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:08.110893 9403 raft_consensus.cc:2454] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 0.
I20240501 07:59:08.111080 7207 leader_election.cc:302] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:08.111295 18355 raft_consensus.cc:2781] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.111352 18355 raft_consensus.cc:482] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.111390 18355 raft_consensus.cc:3037] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.112396 18355 raft_consensus.cc:504] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.112718 18355 leader_election.cc:288] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:08.114817 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd8c956cd084349af9d6a7c2b5fc7a1" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:08.114948 9403 raft_consensus.cc:3037] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.116053 9403 raft_consensus.cc:2454] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:08.116587 7207 leader_election.cc:302] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:08.116762 18355 raft_consensus.cc:2781] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.116731 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd8c956cd084349af9d6a7c2b5fc7a1" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:08.116835 18355 raft_consensus.cc:686] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:08.116849 9270 raft_consensus.cc:3037] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.116954 18355 consensus_queue.cc:237] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.117972 9270 raft_consensus.cc:2454] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:08.120280 6736 catalog_manager.cc:5484] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:08.122170 9270 consensus_queue.cc:237] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.123623 9403 raft_consensus.cc:1261] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.123678 9666 raft_consensus.cc:1261] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.124032 18532 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:08.124235 17449 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:08.124842 9205 consensus_peers.cc:485] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7c14004fdb24e96bacbf8589518f3b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.125981 18532 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:08.126530 9403 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:08.129752 9666 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:08.131906 6736 catalog_manager.cc:5484] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index -1 to 2, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:08.152947 6731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:08.159413 19005 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.159843 19006 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.164433 19007 tablet.cc:1722] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.167083 6731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=031f4d129c7a47029a63689d2855d2a5] alter complete (version 3)
I20240501 07:59:08.171860 7270 consensus_queue.cc:237] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.173236 7138 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:08.173260 9666 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 07:59:08.173399 7208 consensus_peers.cc:485] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7dcdd38eff43fe9fff9081ee644c8d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.173616 18355 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:08.174368 18355 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:08.176295 18355 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.177353 7138 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.180002 9666 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.181844 6736 catalog_manager.cc:5484] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:08.272547 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:08.272774 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:08.276118 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 9ded6f45a3734c02a454b1317b6bfa20
I20240501 07:59:08.277112 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:58254
I20240501 07:59:08.277429 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:08.277887 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded6f45a3734c02a454b1317b6bfa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:08.278643 19173 tablet_replica.cc:330] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:08.279687 19172 tablet_replica.cc:330] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:59:08.279846 19172 raft_consensus.cc:2229] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.279968 19172 raft_consensus.cc:2258] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.280200 19174 tablet_replica.cc:330] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:08.280326 19174 raft_consensus.cc:2229] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.280805 19172 ts_tablet_manager.cc:1875] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.279707 19173 raft_consensus.cc:2229] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.282064 19173 raft_consensus.cc:2258] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.282852 19173 ts_tablet_manager.cc:1875] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.283849 19172 ts_tablet_manager.cc:1888] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.283932 19172 log.cc:1196] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ded6f45a3734c02a454b1317b6bfa20
I20240501 07:59:08.284386 19172 ts_tablet_manager.cc:1909] T 9ded6f45a3734c02a454b1317b6bfa20 P 15ab96f6477b4291932409b80af07213: Deleting consensus metadata
I20240501 07:59:08.284889 19174 raft_consensus.cc:2258] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.285707 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table_new_name [id=031f4d129c7a47029a63689d2855d2a5]) successfully deleted
I20240501 07:59:08.286296 19174 ts_tablet_manager.cc:1875] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.286509 19173 ts_tablet_manager.cc:1888] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.286608 19173 log.cc:1196] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9ded6f45a3734c02a454b1317b6bfa20
I20240501 07:59:08.287202 19173 ts_tablet_manager.cc:1909] T 9ded6f45a3734c02a454b1317b6bfa20 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:59:08.290328 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table_new_name [id=031f4d129c7a47029a63689d2855d2a5]) successfully deleted
I20240501 07:59:08.300076 19174 ts_tablet_manager.cc:1888] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.300284 19174 log.cc:1196] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9ded6f45a3734c02a454b1317b6bfa20
I20240501 07:59:08.300619 19174 ts_tablet_manager.cc:1909] T 9ded6f45a3734c02a454b1317b6bfa20 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:08.301316 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 9ded6f45a3734c02a454b1317b6bfa20 (table rebalancer_dynamic_table_new_name [id=031f4d129c7a47029a63689d2855d2a5]) successfully deleted
I20240501 07:59:08.334739 9644 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:08.336621 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:08.337385 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.350633 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.355677 6990 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.371970 9507 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:08.379959 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:08.386713 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 12766879febc498ebcbb25b2954632e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]), partition=
I20240501 07:59:08.386983 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12766879febc498ebcbb25b2954632e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.389796 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 12766879febc498ebcbb25b2954632e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]), partition=
I20240501 07:59:08.390007 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12766879febc498ebcbb25b2954632e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.392122 19192 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:08.393364 19192 tablet_bootstrap.cc:654] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.393463 17598 consensus_queue.cc:1035] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.394549 19190 ts_tablet_manager.cc:902] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:08.395051 19190 tablet_copy_client.cc:308] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:08.395687 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 12766879febc498ebcbb25b2954632e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]), partition=
I20240501 07:59:08.395958 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12766879febc498ebcbb25b2954632e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.395938 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 8a4ad48cbfb947f4aec462093006f6c7 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:54625)
I20240501 07:59:08.396063 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 8a4ad48cbfb947f4aec462093006f6c7 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:54625: session id = 9a3888360d4d4ace96b6b909c160bfc2-8a4ad48cbfb947f4aec462093006f6c7
I20240501 07:59:08.396993 9555 tablet_copy_source_session.cc:205] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.398192 19190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4ad48cbfb947f4aec462093006f6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.399078 18639 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:08.399080 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:08.400640 18639 tablet_bootstrap.cc:654] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.408361 18639 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:08.408466 18639 ts_tablet_manager.cc:1367] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 07:59:08.409282 18639 raft_consensus.cc:348] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.409402 18639 raft_consensus.cc:374] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.409435 18639 raft_consensus.cc:729] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.409556 18639 consensus_queue.cc:260] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.409855 18639 ts_tablet_manager.cc:1398] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.410660 7121 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:08.426697 19192 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:08.426833 19192 ts_tablet_manager.cc:1367] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240501 07:59:08.427623 19192 raft_consensus.cc:348] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.427755 19192 raft_consensus.cc:374] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.427794 19192 raft_consensus.cc:729] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.427943 19192 consensus_queue.cc:260] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.428290 19192 ts_tablet_manager.cc:1398] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.431242 19191 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:08.432301 19191 tablet_bootstrap.cc:654] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.432585 19190 tablet_copy_client.cc:783] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.432735 19190 tablet_copy_client.cc:647] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.439644 17891 consensus_queue.cc:1035] T 1cebd607453c4cb6a0d4762dfddc1464 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.447741 19190 tablet_copy_client.cc:523] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.451277 19190 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:08.454234 19191 tablet_bootstrap.cc:492] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:08.454332 19191 ts_tablet_manager.cc:1367] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 07:59:08.454735 19191 raft_consensus.cc:348] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.454818 19191 raft_consensus.cc:374] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.454838 19191 raft_consensus.cc:729] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.454916 19191 consensus_queue.cc:260] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.455209 19191 ts_tablet_manager.cc:1398] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.465121 19190 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.465500 19190 tablet_bootstrap.cc:492] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:08.465922 19190 ts_tablet_manager.cc:1367] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 07:59:08.467031 19190 raft_consensus.cc:348] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:08.467965 19190 raft_consensus.cc:729] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.468153 19190 consensus_queue.cc:260] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:08.472273 19190 ts_tablet_manager.cc:1398] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:08.474740 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-8a4ad48cbfb947f4aec462093006f6c7 received from {username='slave'} at 127.0.99.130:54625
I20240501 07:59:08.474823 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-8a4ad48cbfb947f4aec462093006f6c7 on tablet 8a4ad48cbfb947f4aec462093006f6c7 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:08.474606 19211 ts_tablet_manager.cc:902] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:08.475967 19211 tablet_copy_client.cc:308] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:08.476169 19217 ts_tablet_manager.cc:902] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:08.476531 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet d30d4d6ddf3a40989b10cf5c10ef7082 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:50509)
I20240501 07:59:08.476651 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet d30d4d6ddf3a40989b10cf5c10ef7082 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:50509: session id = 1a9029bee6a144938f68b363e4cca162-d30d4d6ddf3a40989b10cf5c10ef7082
I20240501 07:59:08.476682 19217 tablet_copy_client.cc:308] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:59:08.477656 9555 tablet_copy_source_session.cc:205] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.477669 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet 010c12e2cc194cdb87ff9299a8e42a77 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:42663)
I20240501 07:59:08.477771 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet 010c12e2cc194cdb87ff9299a8e42a77 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:42663: session id = 15ab96f6477b4291932409b80af07213-010c12e2cc194cdb87ff9299a8e42a77
I20240501 07:59:08.478582 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30d4d6ddf3a40989b10cf5c10ef7082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.479130 9414 tablet_copy_source_session.cc:205] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.479262 19219 ts_tablet_manager.cc:902] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:08.480247 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010c12e2cc194cdb87ff9299a8e42a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.481825 19211 tablet_copy_client.cc:783] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.482069 19211 tablet_copy_client.cc:647] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.482252 19217 tablet_copy_client.cc:783] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.482416 19217 tablet_copy_client.cc:647] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.479876 19219 tablet_copy_client.cc:308] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:08.483179 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 9051ba90bad3430fa2e128c88ec7b6f4 from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:44297)
I20240501 07:59:08.483289 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 9051ba90bad3430fa2e128c88ec7b6f4 from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:44297: session id = ac6c8bc1b74b483791e262abee7862cf-9051ba90bad3430fa2e128c88ec7b6f4
I20240501 07:59:08.483646 19217 tablet_copy_client.cc:523] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.484254 7292 tablet_copy_source_session.cc:205] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.484670 19217 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:08.485332 19219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051ba90bad3430fa2e128c88ec7b6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.485450 19211 tablet_copy_client.cc:523] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.486783 19211 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:08.488019 19217 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.488270 19217 tablet_bootstrap.cc:492] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:59:08.488431 19217 ts_tablet_manager.cc:1367] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:59:08.488879 19217 raft_consensus.cc:348] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.488999 19217 raft_consensus.cc:729] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.489154 19217 consensus_queue.cc:260] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.490196 19217 ts_tablet_manager.cc:1398] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:08.490629 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-010c12e2cc194cdb87ff9299a8e42a77 received from {username='slave'} at 127.0.99.132:42663
I20240501 07:59:08.490711 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 15ab96f6477b4291932409b80af07213-010c12e2cc194cdb87ff9299a8e42a77 on tablet 010c12e2cc194cdb87ff9299a8e42a77 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:59:08.491617 19219 tablet_copy_client.cc:783] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.491778 19219 tablet_copy_client.cc:647] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.493772 19219 tablet_copy_client.cc:523] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.494374 19211 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.494782 19211 tablet_bootstrap.cc:492] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:08.494943 19211 ts_tablet_manager.cc:1367] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 07:59:08.495298 19219 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:08.495416 19211 raft_consensus.cc:348] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.495534 19211 raft_consensus.cc:729] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.495715 19211 consensus_queue.cc:260] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.497107 19211 ts_tablet_manager.cc:1398] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.497618 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-d30d4d6ddf3a40989b10cf5c10ef7082 received from {username='slave'} at 127.0.99.129:50509
I20240501 07:59:08.497706 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-d30d4d6ddf3a40989b10cf5c10ef7082 on tablet d30d4d6ddf3a40989b10cf5c10ef7082 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:08.500629 19235 ts_tablet_manager.cc:902] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:08.501158 19235 tablet_copy_client.cc:308] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:08.501715 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 81e4af3cf8f044e2be0f8e3195f8c084 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:50509)
I20240501 07:59:08.501816 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 81e4af3cf8f044e2be0f8e3195f8c084 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:50509: session id = 1a9029bee6a144938f68b363e4cca162-81e4af3cf8f044e2be0f8e3195f8c084
I20240501 07:59:08.502806 9555 tablet_copy_source_session.cc:205] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.503795 19235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4af3cf8f044e2be0f8e3195f8c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.514207 18938 consensus_queue.cc:1035] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.527257 19235 tablet_copy_client.cc:783] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.527539 19235 tablet_copy_client.cc:647] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.529737 19235 tablet_copy_client.cc:523] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.531050 19235 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:08.536101 19219 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.536262 19235 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.536406 19219 tablet_bootstrap.cc:492] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:59:08.536512 19235 tablet_bootstrap.cc:492] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:08.536579 19219 ts_tablet_manager.cc:1367] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.041s user 0.028s sys 0.000s
I20240501 07:59:08.536675 19235 ts_tablet_manager.cc:1367] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 07:59:08.536985 19219 raft_consensus.cc:348] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.537086 19219 raft_consensus.cc:729] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.537046 19235 raft_consensus.cc:348] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.537173 19235 raft_consensus.cc:729] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.537230 19219 consensus_queue.cc:260] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.537297 19235 consensus_queue.cc:260] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.537596 19235 ts_tablet_manager.cc:1398] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.538291 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-81e4af3cf8f044e2be0f8e3195f8c084 received from {username='slave'} at 127.0.99.129:50509
I20240501 07:59:08.538363 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-81e4af3cf8f044e2be0f8e3195f8c084 on tablet 81e4af3cf8f044e2be0f8e3195f8c084 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:08.538784 19219 ts_tablet_manager.cc:1398] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:08.539364 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-9051ba90bad3430fa2e128c88ec7b6f4 received from {username='slave'} at 127.0.99.135:44297
I20240501 07:59:08.539464 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-9051ba90bad3430fa2e128c88ec7b6f4 on tablet 9051ba90bad3430fa2e128c88ec7b6f4 with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:08.544076 18937 consensus_queue.cc:1035] T 1bd8c956cd084349af9d6a7c2b5fc7a1 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.554030 18393 consensus_queue.cc:1035] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.558177 18938 consensus_queue.cc:1035] T eae3ef1d35a2437692eed3696ebd6649 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.583339 19276 ts_tablet_manager.cc:902] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:08.584087 19276 tablet_copy_client.cc:308] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:08.584611 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 4e7f613e16c245de83fde8c277b8d350 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:49897)
I20240501 07:59:08.584741 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 4e7f613e16c245de83fde8c277b8d350 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:49897: session id = fa6964452854415a85ad72f8639768d6-4e7f613e16c245de83fde8c277b8d350
I20240501 07:59:08.585820 7292 tablet_copy_source_session.cc:205] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.587738 19219 ts_tablet_manager.cc:902] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:08.588160 19219 tablet_copy_client.cc:308] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:08.594024 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet ab7dcdd38eff43fe9fff9081ee644c8d from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:44297)
I20240501 07:59:08.594151 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet ab7dcdd38eff43fe9fff9081ee644c8d from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:44297: session id = ac6c8bc1b74b483791e262abee7862cf-ab7dcdd38eff43fe9fff9081ee644c8d
I20240501 07:59:08.595113 7292 tablet_copy_source_session.cc:205] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.596426 19276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f613e16c245de83fde8c277b8d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.596661 19219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7dcdd38eff43fe9fff9081ee644c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.599882 19219 tablet_copy_client.cc:783] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.600055 19219 tablet_copy_client.cc:647] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.604573 19219 tablet_copy_client.cc:523] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.605993 19219 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:08.612162 19276 tablet_copy_client.cc:783] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.612362 19276 tablet_copy_client.cc:647] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.613741 19276 tablet_copy_client.cc:523] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.615181 19276 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:08.642659 18564 raft_consensus.cc:482] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.642791 18564 raft_consensus.cc:504] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.643247 18564 leader_election.cc:288] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:08.643651 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12766879febc498ebcbb25b2954632e4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:08.643797 9271 raft_consensus.cc:2454] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:08.644021 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12766879febc498ebcbb25b2954632e4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:08.644151 9535 raft_consensus.cc:2454] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:08.644256 9601 leader_election.cc:302] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:08.644459 18564 raft_consensus.cc:2781] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.644516 18564 raft_consensus.cc:482] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.644548 18564 raft_consensus.cc:3037] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.645634 18564 raft_consensus.cc:504] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.645990 18564 leader_election.cc:288] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:08.646646 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12766879febc498ebcbb25b2954632e4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:08.646672 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12766879febc498ebcbb25b2954632e4" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:08.646731 19276 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.646752 9535 raft_consensus.cc:3037] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.646767 9271 raft_consensus.cc:3037] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.647075 19276 tablet_bootstrap.cc:492] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:59:08.647258 19276 ts_tablet_manager.cc:1367] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.005s
I20240501 07:59:08.647768 9271 raft_consensus.cc:2454] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:08.647768 9535 raft_consensus.cc:2454] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:08.647729 19276 raft_consensus.cc:348] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.647845 19276 raft_consensus.cc:729] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.647969 19276 consensus_queue.cc:260] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:08.648061 9602 leader_election.cc:302] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf; no voters:
I20240501 07:59:08.648528 18564 raft_consensus.cc:2781] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.648633 18564 raft_consensus.cc:686] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:08.648762 18564 consensus_queue.cc:237] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.649194 19276 ts_tablet_manager.cc:1398] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.650547 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-4e7f613e16c245de83fde8c277b8d350 received from {username='slave'} at 127.0.99.134:49897
I20240501 07:59:08.650622 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session fa6964452854415a85ad72f8639768d6-4e7f613e16c245de83fde8c277b8d350 on tablet 4e7f613e16c245de83fde8c277b8d350 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:59:08.651620 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:08.652302 6735 catalog_manager.cc:5484] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:08.659662 19217 ts_tablet_manager.cc:902] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:08.660136 19217 tablet_copy_client.cc:308] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:59:08.660605 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet f7c14004fdb24e96bacbf8589518f3b1 from peer 15ab96f6477b4291932409b80af07213 ({username='slave'} at 127.0.99.132:44575)
I20240501 07:59:08.660719 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet f7c14004fdb24e96bacbf8589518f3b1 from peer 15ab96f6477b4291932409b80af07213 at {username='slave'} at 127.0.99.132:44575: session id = 15ab96f6477b4291932409b80af07213-f7c14004fdb24e96bacbf8589518f3b1
I20240501 07:59:08.661717 9291 tablet_copy_source_session.cc:205] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.662775 19217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c14004fdb24e96bacbf8589518f3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.663595 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 78b1050d1679407f999a596e798c0cb3 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.663882 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b1050d1679407f999a596e798c0cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.664741 19219 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.664907 19217 tablet_copy_client.cc:783] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.665005 19219 tablet_bootstrap.cc:492] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:59:08.665045 19217 tablet_copy_client.cc:647] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.665180 19219 ts_tablet_manager.cc:1367] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.059s user 0.024s sys 0.003s
I20240501 07:59:08.665675 19219 raft_consensus.cc:348] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.665783 19219 raft_consensus.cc:729] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.665925 19219 consensus_queue.cc:260] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:08.666218 19219 ts_tablet_manager.cc:1398] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.666580 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 78b1050d1679407f999a596e798c0cb3 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.666829 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b1050d1679407f999a596e798c0cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.666837 6989 tablet_service.cc:1467] Processing CreateTablet for tablet 548de37522934cb5a29bf1014baa7edd (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.667172 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548de37522934cb5a29bf1014baa7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.667255 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-ab7dcdd38eff43fe9fff9081ee644c8d received from {username='slave'} at 127.0.99.135:44297
I20240501 07:59:08.667330 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-ab7dcdd38eff43fe9fff9081ee644c8d on tablet ab7dcdd38eff43fe9fff9081ee644c8d with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:08.668550 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 3150b34bfbad4ffda16384624ce766b7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.668815 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150b34bfbad4ffda16384624ce766b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.671095 19295 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:08.672278 7121 tablet_service.cc:1467] Processing CreateTablet for tablet 548de37522934cb5a29bf1014baa7edd (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.672531 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548de37522934cb5a29bf1014baa7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.673187 19293 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:08.673409 19294 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:08.676167 19295 tablet_bootstrap.cc:654] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.676767 9250 tablet_service.cc:1467] Processing CreateTablet for tablet 78b1050d1679407f999a596e798c0cb3 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.677094 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b1050d1679407f999a596e798c0cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.677026 6857 tablet_service.cc:1467] Processing CreateTablet for tablet 3150b34bfbad4ffda16384624ce766b7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.677302 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150b34bfbad4ffda16384624ce766b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.677381 19295 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:08.677453 19295 ts_tablet_manager.cc:1367] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:08.677881 19295 raft_consensus.cc:348] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.677981 19295 raft_consensus.cc:374] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.678020 19295 raft_consensus.cc:729] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.678131 19295 consensus_queue.cc:260] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.678392 19295 ts_tablet_manager.cc:1398] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.678805 19294 tablet_bootstrap.cc:654] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.679850 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 3150b34bfbad4ffda16384624ce766b7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.680105 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3150b34bfbad4ffda16384624ce766b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.680469 6858 tablet_service.cc:1467] Processing CreateTablet for tablet 548de37522934cb5a29bf1014baa7edd (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=62963f98d4664429b704b5fca3bb108a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.680691 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548de37522934cb5a29bf1014baa7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.681982 19293 tablet_bootstrap.cc:654] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.682418 19191 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:08.683100 19293 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:08.683185 19293 ts_tablet_manager.cc:1367] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 07:59:08.683633 19293 raft_consensus.cc:348] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.683743 19293 raft_consensus.cc:374] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.683779 19293 raft_consensus.cc:729] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.683888 19293 consensus_queue.cc:260] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.684176 19293 ts_tablet_manager.cc:1398] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.684346 19293 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:08.684654 19191 tablet_bootstrap.cc:654] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.685310 19293 tablet_bootstrap.cc:654] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.688333 18639 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:08.689252 18639 tablet_bootstrap.cc:654] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.691449 19191 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:08.691540 19191 ts_tablet_manager.cc:1367] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:08.691990 19191 raft_consensus.cc:348] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.692101 19191 raft_consensus.cc:374] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.692134 19191 raft_consensus.cc:729] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.692265 19191 consensus_queue.cc:260] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.692570 19191 ts_tablet_manager.cc:1398] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.695755 19294 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:08.695878 19294 ts_tablet_manager.cc:1367] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 07:59:08.696353 19294 raft_consensus.cc:348] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.696487 19294 raft_consensus.cc:374] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.696520 19294 raft_consensus.cc:729] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.696632 19294 consensus_queue.cc:260] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.696954 19294 ts_tablet_manager.cc:1398] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.697239 19294 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:08.698174 19294 tablet_bootstrap.cc:654] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.698375 19217 tablet_copy_client.cc:523] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.698530 19299 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:08.699961 19217 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:08.700122 19293 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:08.700202 19293 ts_tablet_manager.cc:1367] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240501 07:59:08.700677 19293 raft_consensus.cc:348] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.700790 19293 raft_consensus.cc:374] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.700851 19293 raft_consensus.cc:729] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.700966 19293 consensus_queue.cc:260] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.701224 19293 ts_tablet_manager.cc:1398] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.707844 19299 tablet_bootstrap.cc:654] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.709555 18639 tablet_bootstrap.cc:492] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:08.709659 18639 ts_tablet_manager.cc:1367] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 07:59:08.710101 18639 raft_consensus.cc:348] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.710234 18639 raft_consensus.cc:374] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.710274 18639 raft_consensus.cc:729] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.710388 18639 consensus_queue.cc:260] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.710665 18639 ts_tablet_manager.cc:1398] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.712251 19294 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:08.712347 19294 ts_tablet_manager.cc:1367] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:08.713570 19294 raft_consensus.cc:348] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.713914 19294 raft_consensus.cc:374] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.714015 19294 raft_consensus.cc:729] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.714319 19294 consensus_queue.cc:260] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.719069 19217 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.719348 19217 tablet_bootstrap.cc:492] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Bootstrap complete.
I20240501 07:59:08.719503 19217 ts_tablet_manager.cc:1367] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240501 07:59:08.719949 19217 raft_consensus.cc:348] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.720185 19294 ts_tablet_manager.cc:1398] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:08.720547 19299 tablet_bootstrap.cc:492] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:08.720997 19299 ts_tablet_manager.cc:1367] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240501 07:59:08.721753 19299 raft_consensus.cc:348] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.722517 19217 raft_consensus.cc:729] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.722661 19217 consensus_queue.cc:260] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.722949 19217 ts_tablet_manager.cc:1398] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:08.724684 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session 15ab96f6477b4291932409b80af07213-f7c14004fdb24e96bacbf8589518f3b1 received from {username='slave'} at 127.0.99.132:44575
I20240501 07:59:08.724761 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session 15ab96f6477b4291932409b80af07213-f7c14004fdb24e96bacbf8589518f3b1 on tablet f7c14004fdb24e96bacbf8589518f3b1 with peer 15ab96f6477b4291932409b80af07213
I20240501 07:59:08.721922 19299 raft_consensus.cc:374] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.730160 19299 raft_consensus.cc:729] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.731160 19299 consensus_queue.cc:260] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.734526 19299 ts_tablet_manager.cc:1398] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 07:59:08.735512 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:08.735759 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.735993 19299 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:08.737097 19299 tablet_bootstrap.cc:654] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.746629 17598 raft_consensus.cc:482] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.746747 17598 raft_consensus.cc:504] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.747164 17598 leader_election.cc:288] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:08.747581 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548de37522934cb5a29bf1014baa7edd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:08.747705 7010 raft_consensus.cc:2454] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.748116 7077 leader_election.cc:302] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.748363 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548de37522934cb5a29bf1014baa7edd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:08.748476 6877 raft_consensus.cc:2454] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.748505 17598 raft_consensus.cc:2781] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.748555 17598 raft_consensus.cc:482] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.748587 17598 raft_consensus.cc:3037] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.749626 17598 raft_consensus.cc:504] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.749943 17598 leader_election.cc:288] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:08.750941 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548de37522934cb5a29bf1014baa7edd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:08.751081 7010 raft_consensus.cc:3037] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.751654 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548de37522934cb5a29bf1014baa7edd" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:08.751760 6877 raft_consensus.cc:3037] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.752079 7010 raft_consensus.cc:2454] T 548de37522934cb5a29bf1014baa7edd P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.752852 6877 raft_consensus.cc:2454] T 548de37522934cb5a29bf1014baa7edd P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.753274 7076 leader_election.cc:302] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.753538 17891 raft_consensus.cc:2781] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.753618 17891 raft_consensus.cc:686] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:08.753732 17891 consensus_queue.cc:237] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:08.759116 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.766121 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:08.773640 6732 catalog_manager.cc:5484] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:08.775372 7121 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:08.779484 19299 tablet_bootstrap.cc:492] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:08.779650 19299 ts_tablet_manager.cc:1367] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20240501 07:59:08.781540 19299 raft_consensus.cc:348] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.781764 19299 raft_consensus.cc:374] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.781849 19299 raft_consensus.cc:729] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.782059 9507 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:08.782356 19299 consensus_queue.cc:260] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.782408 7010 raft_consensus.cc:1203] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:08.782737 19299 ts_tablet_manager.cc:1398] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:08.791705 19127 raft_consensus.cc:482] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.791808 19127 raft_consensus.cc:504] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.792160 19127 leader_election.cc:288] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:08.792732 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3150b34bfbad4ffda16384624ce766b7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:08.792863 6877 raft_consensus.cc:2454] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:08.795351 9601 leader_election.cc:302] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:08.795853 19127 raft_consensus.cc:2781] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.795918 19127 raft_consensus.cc:482] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.795962 19127 raft_consensus.cc:3037] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.796003 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3150b34bfbad4ffda16384624ce766b7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:08.796160 7270 raft_consensus.cc:2454] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 0.
I20240501 07:59:08.796339 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:08.797222 19127 raft_consensus.cc:504] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.797600 19127 leader_election.cc:288] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:08.798712 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3150b34bfbad4ffda16384624ce766b7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:08.798823 6878 raft_consensus.cc:3037] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.799803 6878 raft_consensus.cc:2454] T 3150b34bfbad4ffda16384624ce766b7 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:08.800172 9601 leader_election.cc:302] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460; no voters:
I20240501 07:59:08.804242 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:08.807164 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3150b34bfbad4ffda16384624ce766b7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:08.807292 7270 raft_consensus.cc:3037] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.807965 19127 raft_consensus.cc:2781] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.808056 19127 raft_consensus.cc:686] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:08.808194 19127 consensus_queue.cc:237] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:08.808339 7270 raft_consensus.cc:2454] T 3150b34bfbad4ffda16384624ce766b7 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:08.815488 6730 catalog_manager.cc:5484] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:08.832141 9271 raft_consensus.cc:1261] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.832662 19127 consensus_queue.cc:1035] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.836213 6735 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]). Expected version 1 got 0
I20240501 07:59:08.837633 19323 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.837767 9535 raft_consensus.cc:1261] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.838121 19323 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:08.838261 18564 consensus_queue.cc:1035] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.839473 9644 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:08.844219 6730 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]). Expected version 1 got 0
I20240501 07:59:08.844451 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c] alter complete (version 1)
I20240501 07:59:08.846824 6735 catalog_manager.cc:5576] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013) does not have the latest schema for tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c]). Expected version 1 got 0
I20240501 07:59:08.851629 19322 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.852012 19322 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:08.854800 19326 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.856211 19326 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:08.873450 17891 raft_consensus.cc:482] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.873564 17891 raft_consensus.cc:504] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.873988 17891 leader_election.cc:288] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:08.874336 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b1050d1679407f999a596e798c0cb3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:08.874481 9271 raft_consensus.cc:2454] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.875221 7076 leader_election.cc:302] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.875278 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b1050d1679407f999a596e798c0cb3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:08.875432 7009 raft_consensus.cc:2454] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:08.876230 17891 raft_consensus.cc:2781] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.876296 17891 raft_consensus.cc:482] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.876344 17891 raft_consensus.cc:3037] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.877277 17891 raft_consensus.cc:504] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.877923 17891 leader_election.cc:288] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:08.877923 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b1050d1679407f999a596e798c0cb3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:08.878047 7009 raft_consensus.cc:3037] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.878211 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b1050d1679407f999a596e798c0cb3" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:08.878330 9271 raft_consensus.cc:3037] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.879179 7009 raft_consensus.cc:2454] T 78b1050d1679407f999a596e798c0cb3 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.879289 9271 raft_consensus.cc:2454] T 78b1050d1679407f999a596e798c0cb3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:08.879501 7077 leader_election.cc:302] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:08.879808 17891 raft_consensus.cc:2781] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.879901 17891 raft_consensus.cc:686] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:08.880110 17891 consensus_queue.cc:237] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:08.882936 6735 catalog_manager.cc:5484] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:08.910782 6878 raft_consensus.cc:1203] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:08.931526 9534 raft_consensus.cc:1203] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:08.957594 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:08.963177 19322 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.963515 19322 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:08.964385 19326 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.964704 19326 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: MemRowSet was empty: no flush needed.
I20240501 07:59:08.970327 19323 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.970624 19323 tablet.cc:1620] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:08.971098 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=d68001bcbd2e473482b4693b33ea4f1c] alter complete (version 2)
I20240501 07:59:09.001920 6878 raft_consensus.cc:1203] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:09.020632 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.024932 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:09.030186 9644 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:09.030207 7270 raft_consensus.cc:1203] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:09.034600 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.041230 9507 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:09.048712 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.057385 7270 raft_consensus.cc:1203] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:09.058622 7121 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:09.080830 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:09.099326 19322 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.100247 19323 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.103734 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:09.105469 19326 tablet.cc:1722] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.106081 6730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table_new_name [id=d68001bcbd2e473482b4693b33ea4f1c] alter complete (version 3)
I20240501 07:59:09.114084 9401 raft_consensus.cc:1203] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:09.139072 9534 raft_consensus.cc:1203] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:09.157018 19369 consensus_queue.cc:1035] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.165553 19369 consensus_queue.cc:1035] T 548de37522934cb5a29bf1014baa7edd P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.214679 6728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:09.214836 6728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:09.218135 6728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: Sending DeleteTablet for 3 replicas of tablet 12766879febc498ebcbb25b2954632e4
I20240501 07:59:09.218649 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 12766879febc498ebcbb25b2954632e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:09.218703 9507 tablet_service.cc:1514] Processing DeleteTablet for tablet 12766879febc498ebcbb25b2954632e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:09.218935 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 12766879febc498ebcbb25b2954632e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:09.218986 19427 tablet_replica.cc:330] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:09.219239 19427 raft_consensus.cc:2229] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.219616 19427 raft_consensus.cc:2258] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.220253 19427 ts_tablet_manager.cc:1875] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.225862 19427 ts_tablet_manager.cc:1888] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.225948 19427 log.cc:1196] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/12766879febc498ebcbb25b2954632e4
I20240501 07:59:09.226234 19427 ts_tablet_manager.cc:1909] T 12766879febc498ebcbb25b2954632e4 P 7fa3a88c25964445975bb47e0feed460: Deleting consensus metadata
I20240501 07:59:09.228060 19428 tablet_replica.cc:330] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:09.228307 19428 raft_consensus.cc:2229] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.229136 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table_new_name [id=d68001bcbd2e473482b4693b33ea4f1c]) successfully deleted
I20240501 07:59:09.229159 19428 raft_consensus.cc:2258] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.231380 19428 ts_tablet_manager.cc:1875] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.234076 19428 ts_tablet_manager.cc:1888] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.235378 19428 log.cc:1196] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/12766879febc498ebcbb25b2954632e4
I20240501 07:59:09.235733 19428 ts_tablet_manager.cc:1909] T 12766879febc498ebcbb25b2954632e4 P ac6c8bc1b74b483791e262abee7862cf: Deleting consensus metadata
I20240501 07:59:09.240578 19429 tablet_replica.cc:330] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:09.240832 19429 raft_consensus.cc:2229] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.241031 19429 raft_consensus.cc:2258] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.241686 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table_new_name [id=d68001bcbd2e473482b4693b33ea4f1c]) successfully deleted
I20240501 07:59:09.242234 19429 ts_tablet_manager.cc:1875] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.245371 19429 ts_tablet_manager.cc:1888] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.245519 19429 log.cc:1196] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/12766879febc498ebcbb25b2954632e4
I20240501 07:59:09.247318 19429 ts_tablet_manager.cc:1909] T 12766879febc498ebcbb25b2954632e4 P 49adb9e99822494196ed22044bfed81f: Deleting consensus metadata
I20240501 07:59:09.248096 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 12766879febc498ebcbb25b2954632e4 (table rebalancer_dynamic_table_new_name [id=d68001bcbd2e473482b4693b33ea4f1c]) successfully deleted
I20240501 07:59:09.263059 18564 consensus_queue.cc:1035] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.266553 18564 consensus_queue.cc:1035] T 3150b34bfbad4ffda16384624ce766b7 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.270505 19369 consensus_queue.cc:1035] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.275734 17598 consensus_queue.cc:1035] T 78b1050d1679407f999a596e798c0cb3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.312074 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.312434 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.315658 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.321233 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:09.334735 9507 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:09.337558 9381 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce273cfa1945bdbe969faa0e864037 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4]), partition=
I20240501 07:59:09.337810 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce273cfa1945bdbe969faa0e864037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.340020 9644 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce273cfa1945bdbe969faa0e864037 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4]), partition=
I20240501 07:59:09.340262 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce273cfa1945bdbe969faa0e864037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.340250 9250 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce273cfa1945bdbe969faa0e864037 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4]), partition=
I20240501 07:59:09.340476 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce273cfa1945bdbe969faa0e864037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.343875 19447 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:09.343984 6857 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:09.345286 19447 tablet_bootstrap.cc:654] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.352355 19448 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:09.353298 19448 tablet_bootstrap.cc:654] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.365368 19448 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: No bootstrap required, opened a new log
I20240501 07:59:09.365564 19448 ts_tablet_manager.cc:1367] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:09.366564 19448 raft_consensus.cc:348] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.366890 9381 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:09.367121 19448 raft_consensus.cc:374] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.367235 19448 raft_consensus.cc:729] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.367523 7121 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:09.370074 19448 consensus_queue.cc:260] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.371519 19448 ts_tablet_manager.cc:1398] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:09.374507 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:09.377141 19446 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:09.378157 19446 tablet_bootstrap.cc:654] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.397269 6729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58352:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:09.406738 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 604cff2480b942be85044096cdf95e7f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.407071 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604cff2480b942be85044096cdf95e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.407478 9507 tablet_service.cc:1467] Processing CreateTablet for tablet 604cff2480b942be85044096cdf95e7f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.407560 9515 tablet_service.cc:1467] Processing CreateTablet for tablet e1d646c521664e1995b56a40f0758e66 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.407752 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604cff2480b942be85044096cdf95e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.408007 6989 tablet_service.cc:1467] Processing CreateTablet for tablet e1626488ca9e423aae41511bd1989889 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.408064 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 604cff2480b942be85044096cdf95e7f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.408282 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1626488ca9e423aae41511bd1989889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.408311 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604cff2480b942be85044096cdf95e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.410622 9382 tablet_service.cc:1467] Processing CreateTablet for tablet e1626488ca9e423aae41511bd1989889 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.410866 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1626488ca9e423aae41511bd1989889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.410887 6857 tablet_service.cc:1467] Processing CreateTablet for tablet e1626488ca9e423aae41511bd1989889 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.411181 6857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1626488ca9e423aae41511bd1989889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.413470 19447 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:09.413571 19447 ts_tablet_manager.cc:1367] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20240501 07:59:09.414234 19447 raft_consensus.cc:348] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.414350 19447 raft_consensus.cc:374] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.414384 19447 raft_consensus.cc:729] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.414454 19446 tablet_bootstrap.cc:492] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:09.414533 19446 ts_tablet_manager.cc:1367] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240501 07:59:09.414506 19447 consensus_queue.cc:260] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.414654 9515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d646c521664e1995b56a40f0758e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.414788 19447 ts_tablet_manager.cc:1398] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.414955 19447 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:09.415874 19447 tablet_bootstrap.cc:654] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.414927 19446 raft_consensus.cc:348] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.415992 19446 raft_consensus.cc:374] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.416038 19446 raft_consensus.cc:729] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.416178 19446 consensus_queue.cc:260] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.416426 19446 ts_tablet_manager.cc:1398] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:09.416571 19446 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:09.417292 7250 tablet_service.cc:1467] Processing CreateTablet for tablet e1d646c521664e1995b56a40f0758e66 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.417577 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d646c521664e1995b56a40f0758e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.418002 19446 tablet_bootstrap.cc:654] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.420339 7121 tablet_service.cc:1467] Processing CreateTablet for tablet e1d646c521664e1995b56a40f0758e66 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=5407894b9bb44fad9c13a5fe20487945]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.420607 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d646c521664e1995b56a40f0758e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.421965 19454 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:09.422899 19454 tablet_bootstrap.cc:654] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.425150 19453 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:09.426762 19447 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460: No bootstrap required, opened a new log
I20240501 07:59:09.426846 19447 ts_tablet_manager.cc:1367] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:09.427292 19447 raft_consensus.cc:348] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.427392 19447 raft_consensus.cc:374] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.427425 19447 raft_consensus.cc:729] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.427533 19447 consensus_queue.cc:260] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.427796 19447 ts_tablet_manager.cc:1398] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.427932 19452 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:09.428851 19452 tablet_bootstrap.cc:654] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.429896 19453 tablet_bootstrap.cc:654] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.432135 19457 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213: Bootstrap starting.
I20240501 07:59:09.432368 19458 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:09.433802 19446 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:09.433925 19446 ts_tablet_manager.cc:1367] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:59:09.434324 19452 tablet_bootstrap.cc:492] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:09.434402 19452 ts_tablet_manager.cc:1367] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:09.434383 19446 raft_consensus.cc:348] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.434504 19446 raft_consensus.cc:374] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.434533 19446 raft_consensus.cc:729] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.434640 19446 consensus_queue.cc:260] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.434916 19446 ts_tablet_manager.cc:1398] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.435088 18254 raft_consensus.cc:482] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.435024 19452 raft_consensus.cc:348] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.435165 18254 raft_consensus.cc:504] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.435220 19452 raft_consensus.cc:374] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.435290 19452 raft_consensus.cc:729] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.435426 19452 consensus_queue.cc:260] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.435597 18254 leader_election.cc:288] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:09.435704 19452 ts_tablet_manager.cc:1398] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.435873 19446 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:09.436219 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce273cfa1945bdbe969faa0e864037" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:09.436388 9270 raft_consensus.cc:2454] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:09.436611 19452 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:09.436607 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce273cfa1945bdbe969faa0e864037" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:09.436725 9666 raft_consensus.cc:2454] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 0.
I20240501 07:59:09.436861 19446 tablet_bootstrap.cc:654] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.437052 9338 leader_election.cc:302] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:09.437491 19452 tablet_bootstrap.cc:654] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.437692 18254 raft_consensus.cc:2781] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.437748 18254 raft_consensus.cc:482] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.437785 18254 raft_consensus.cc:3037] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.438802 18254 raft_consensus.cc:504] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.441601 18254 leader_election.cc:288] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:09.442116 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce273cfa1945bdbe969faa0e864037" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:09.442265 9270 raft_consensus.cc:3037] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.442350 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce273cfa1945bdbe969faa0e864037" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:09.442461 9666 raft_consensus.cc:3037] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.443498 9270 raft_consensus.cc:2454] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:09.443786 9666 raft_consensus.cc:2454] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:09.444084 9338 leader_election.cc:302] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:09.444546 18254 raft_consensus.cc:2781] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.444648 18254 raft_consensus.cc:686] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:09.444770 18254 consensus_queue.cc:237] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:09.444847 19446 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6: No bootstrap required, opened a new log
I20240501 07:59:09.444913 19446 ts_tablet_manager.cc:1367] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:09.445071 19453 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2: No bootstrap required, opened a new log
I20240501 07:59:09.445173 19453 ts_tablet_manager.cc:1367] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:09.445492 19446 raft_consensus.cc:348] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.445606 19446 raft_consensus.cc:374] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.445647 19446 raft_consensus.cc:729] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.445602 19453 raft_consensus.cc:348] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.445719 19453 raft_consensus.cc:374] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.445755 19446 consensus_queue.cc:260] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.445755 19453 raft_consensus.cc:729] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.445938 19453 consensus_queue.cc:260] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.446027 19446 ts_tablet_manager.cc:1398] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.446229 19453 ts_tablet_manager.cc:1398] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.448447 6729 catalog_manager.cc:5484] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:09.449319 19454 tablet_bootstrap.cc:492] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162: No bootstrap required, opened a new log
I20240501 07:59:09.449431 19454 ts_tablet_manager.cc:1367] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
I20240501 07:59:09.449915 19454 raft_consensus.cc:348] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.450016 19454 raft_consensus.cc:374] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.450047 19454 raft_consensus.cc:729] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.450182 19454 consensus_queue.cc:260] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.450492 19454 ts_tablet_manager.cc:1398] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.453002 19452 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf: No bootstrap required, opened a new log
I20240501 07:59:09.453545 19458 tablet_bootstrap.cc:654] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.453958 19457 tablet_bootstrap.cc:654] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.455102 19452 ts_tablet_manager.cc:1367] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240501 07:59:09.455722 19452 raft_consensus.cc:348] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.456249 19452 raft_consensus.cc:374] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.456354 19452 raft_consensus.cc:729] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.456537 19452 consensus_queue.cc:260] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.456951 19452 ts_tablet_manager.cc:1398] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:09.463759 19457 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213: No bootstrap required, opened a new log
I20240501 07:59:09.463457 19458 tablet_bootstrap.cc:492] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be: No bootstrap required, opened a new log
I20240501 07:59:09.463968 19458 ts_tablet_manager.cc:1367] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240501 07:59:09.464085 19457 ts_tablet_manager.cc:1367] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240501 07:59:09.464586 19458 raft_consensus.cc:348] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.464735 19458 raft_consensus.cc:374] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.464773 19458 raft_consensus.cc:729] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.464908 19458 consensus_queue.cc:260] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.465190 19458 ts_tablet_manager.cc:1398] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.465621 19457 raft_consensus.cc:348] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.465715 19457 raft_consensus.cc:374] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.465737 19457 raft_consensus.cc:729] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab96f6477b4291932409b80af07213, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.465842 19457 consensus_queue.cc:260] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.466145 19457 ts_tablet_manager.cc:1398] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.495208 9401 consensus_queue.cc:237] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:09.496945 7271 raft_consensus.cc:1261] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:09.497061 6878 raft_consensus.cc:1261] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:09.497511 19031 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:09.497663 18254 consensus_queue.cc:1035] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:09.499815 19031 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:09.500221 7271 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:09.501681 6878 raft_consensus.cc:2932] T 010c12e2cc194cdb87ff9299a8e42a77 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:09.503396 6737 catalog_manager.cc:5484] T 010c12e2cc194cdb87ff9299a8e42a77 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:09.507308 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 010c12e2cc194cdb87ff9299a8e42a77 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:09.507766 19427 tablet_replica.cc:330] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:09.507931 19427 raft_consensus.cc:2229] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.508075 19427 raft_consensus.cc:2258] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.508569 19427 ts_tablet_manager.cc:1875] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.510658 19427 ts_tablet_manager.cc:1888] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:09.510740 19427 log.cc:1196] T 010c12e2cc194cdb87ff9299a8e42a77 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/010c12e2cc194cdb87ff9299a8e42a77
I20240501 07:59:09.511488 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 010c12e2cc194cdb87ff9299a8e42a77 (table rebalancer_extra_table_4 [id=df6c0d0bfff248f3ab6191c54668af26]) successfully deleted
I20240501 07:59:09.516703 17598 raft_consensus.cc:482] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.516813 17598 raft_consensus.cc:504] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.517215 17598 leader_election.cc:288] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:09.517553 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d646c521664e1995b56a40f0758e66" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:09.517627 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d646c521664e1995b56a40f0758e66" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf" is_pre_election: true
I20240501 07:59:09.517695 7271 raft_consensus.cc:2454] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:09.517773 9534 raft_consensus.cc:2454] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 0.
I20240501 07:59:09.517984 7075 leader_election.cc:302] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:09.518214 17598 raft_consensus.cc:2781] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.518266 17598 raft_consensus.cc:482] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.518317 17598 raft_consensus.cc:3037] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.519470 17598 raft_consensus.cc:504] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.519776 17598 leader_election.cc:288] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:09.520038 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d646c521664e1995b56a40f0758e66" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:09.520143 7271 raft_consensus.cc:3037] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.520133 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d646c521664e1995b56a40f0758e66" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:09.520246 9535 raft_consensus.cc:3037] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.521147 9535 raft_consensus.cc:2454] T e1d646c521664e1995b56a40f0758e66 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:09.521145 7271 raft_consensus.cc:2454] T e1d646c521664e1995b56a40f0758e66 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:09.521438 7077 leader_election.cc:302] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:09.522859 17598 raft_consensus.cc:2781] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.522967 17598 raft_consensus.cc:686] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:09.523154 17598 consensus_queue.cc:237] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:09.527477 6730 catalog_manager.cc:5484] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 0 to 1, leader changed from <none> to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:09.555693 19024 raft_consensus.cc:482] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.555816 19024 raft_consensus.cc:504] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.556260 19024 leader_election.cc:288] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:09.556759 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1626488ca9e423aae41511bd1989889" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:09.556826 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1626488ca9e423aae41511bd1989889" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:09.556902 9401 raft_consensus.cc:2454] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:09.556964 7010 raft_consensus.cc:2454] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 0.
I20240501 07:59:09.557242 6815 leader_election.cc:302] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:09.557476 19024 raft_consensus.cc:2781] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.557525 19024 raft_consensus.cc:482] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.557575 19024 raft_consensus.cc:3037] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.558502 19024 raft_consensus.cc:504] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.558815 19024 leader_election.cc:288] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:09.559547 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1626488ca9e423aae41511bd1989889" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:09.559669 9401 raft_consensus.cc:3037] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.559689 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1626488ca9e423aae41511bd1989889" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:09.559798 7010 raft_consensus.cc:3037] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.560422 9401 raft_consensus.cc:2454] T e1626488ca9e423aae41511bd1989889 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:09.560698 6814 leader_election.cc:302] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:09.560746 7010 raft_consensus.cc:2454] T e1626488ca9e423aae41511bd1989889 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:09.560840 19024 raft_consensus.cc:2781] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.560916 19024 raft_consensus.cc:686] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:09.561046 19024 consensus_queue.cc:237] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.564988 6730 catalog_manager.cc:5484] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:09.570827 6730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:09.576141 9666 raft_consensus.cc:1261] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.576308 9270 raft_consensus.cc:1261] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.576835 19031 consensus_queue.cc:1035] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.577105 19031 consensus_queue.cc:1035] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.580801 6730 catalog_manager.cc:5576] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111) does not have the latest schema for tablet b6ce273cfa1945bdbe969faa0e864037 (table rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4]). Expected version 1 got 0
I20240501 07:59:09.581650 19322 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.581948 19322 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:09.582180 6729 catalog_manager.cc:5576] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797) does not have the latest schema for tablet b6ce273cfa1945bdbe969faa0e864037 (table rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4]). Expected version 1 got 0
I20240501 07:59:09.582808 19511 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.583209 19511 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:09.585366 19323 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.585633 19323 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:09.592185 6729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4] alter complete (version 1)
I20240501 07:59:09.695672 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58352:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:09.702164 19511 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.702488 19511 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P fa6964452854415a85ad72f8639768d6: MemRowSet was empty: no flush needed.
I20240501 07:59:09.702836 19322 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.702992 19323 tablet.cc:1722] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.703181 19322 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P 7fa3a88c25964445975bb47e0feed460: MemRowSet was empty: no flush needed.
I20240501 07:59:09.703284 19323 tablet.cc:1620] T b6ce273cfa1945bdbe969faa0e864037 P 49adb9e99822494196ed22044bfed81f: MemRowSet was empty: no flush needed.
I20240501 07:59:09.714155 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P d467cffe72b54dd1ac420321915f63b7: rebalancer_dynamic_table [id=609a5dbf575147c3b8fb6930a36c1ec4] alter complete (version 2)
I20240501 07:59:09.716113 18653 raft_consensus.cc:482] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.716217 18653 raft_consensus.cc:504] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.716585 18653 leader_election.cc:288] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:09.716893 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:09.717017 9401 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:09.717147 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:09.717283 9666 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 0.
I20240501 07:59:09.717628 9470 leader_election.cc:302] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:09.718111 18653 raft_consensus.cc:2781] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.718168 18653 raft_consensus.cc:482] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.718226 18653 raft_consensus.cc:3037] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.719318 18653 raft_consensus.cc:504] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.719636 18653 leader_election.cc:288] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:09.719888 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:09.719985 9666 raft_consensus.cc:3037] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.719971 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "ac6c8bc1b74b483791e262abee7862cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:09.720075 9401 raft_consensus.cc:3037] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.720922 9666 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:09.720943 9401 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6c8bc1b74b483791e262abee7862cf in term 1.
I20240501 07:59:09.721264 9470 leader_election.cc:302] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:09.721551 18653 raft_consensus.cc:2781] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.721685 18653 raft_consensus.cc:686] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Leader. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:09.721896 18653 consensus_queue.cc:237] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:09.725383 6735 catalog_manager.cc:5484] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6c8bc1b74b483791e262abee7862cf (127.0.99.135). New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:09.752058 6857 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.760079 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:09.765393 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.768663 9515 tablet_service.cc:1429] Tablet server has 27 leaders and 0 scanners
I20240501 07:59:09.770213 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.771829 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:09.797823 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.823108 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:09.854097 9535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082"
dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55682
I20240501 07:59:09.854275 9535 raft_consensus.cc:595] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:09.932641 17891 consensus_queue.cc:1035] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.941581 19369 consensus_queue.cc:1035] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.959508 18995 consensus_queue.cc:1035] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.966077 19024 consensus_queue.cc:1035] T e1626488ca9e423aae41511bd1989889 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.998760 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.999101 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:09.999380 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.006011 6857 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.008492 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.014839 18653 raft_consensus.cc:979] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: : Instructing follower 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:59:10.014943 18653 raft_consensus.cc:1067] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Signalling peer 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:59:10.015904 6876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082"
dest_uuid: "1a9029bee6a144938f68b363e4cca162"
from {username='slave'} at 127.0.99.135:34317
I20240501 07:59:10.016064 6876 raft_consensus.cc:482] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:10.016126 6876 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.017241 6876 raft_consensus.cc:504] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:10.020306 9270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:10.020437 9270 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.021456 9270 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:10.022186 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:10.022333 9403 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.023250 9515 tablet_service.cc:1429] Tablet server has 27 leaders and 0 scanners
I20240501 07:59:10.023339 9403 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:10.024139 6814 leader_election.cc:302] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:10.024547 19024 raft_consensus.cc:2781] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:10.024631 19024 raft_consensus.cc:686] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:10.024771 19024 consensus_queue.cc:237] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:10.027820 6876 leader_election.cc:288] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:10.028999 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30d4d6ddf3a40989b10cf5c10ef7082" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:10.029127 9534 raft_consensus.cc:3032] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:10.029171 9534 raft_consensus.cc:729] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:10.029314 9534 consensus_queue.cc:260] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:10.029495 9534 raft_consensus.cc:3037] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.030536 9534 raft_consensus.cc:2454] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:10.031334 6735 catalog_manager.cc:5484] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:10.044199 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:10.049468 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.157436 19534 consensus_queue.cc:1035] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.162694 19534 consensus_queue.cc:1035] T 604cff2480b942be85044096cdf95e7f P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.270152 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:10.277882 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.281023 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.281286 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:10.287705 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:10.295815 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.296252 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.308997 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.397715 9535 consensus_queue.cc:237] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:10.399176 9271 raft_consensus.cc:1261] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:10.399202 7010 raft_consensus.cc:1261] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:10.399646 19534 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.399869 19013 consensus_queue.cc:1035] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.401752 19534 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:10.402122 9271 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:10.405468 6730 catalog_manager.cc:5484] T 8a4ad48cbfb947f4aec462093006f6c7 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:10.409193 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a4ad48cbfb947f4aec462093006f6c7 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:10.412623 19696 tablet_replica.cc:330] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:10.413137 7010 raft_consensus.cc:2932] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:10.413285 19696 raft_consensus.cc:2229] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.413583 19696 raft_consensus.cc:2258] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.414183 19696 ts_tablet_manager.cc:1875] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.418653 19696 ts_tablet_manager.cc:1888] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:10.418803 19696 log.cc:1196] T 8a4ad48cbfb947f4aec462093006f6c7 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/8a4ad48cbfb947f4aec462093006f6c7
I20240501 07:59:10.419456 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet 8a4ad48cbfb947f4aec462093006f6c7 (table rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]) successfully deleted
I20240501 07:59:10.505234 9271 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:10.505693 18995 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.550733 9403 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:10.552053 18995 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.561774 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.565598 9535 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:10.566825 19024 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.589094 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.593477 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.597349 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:10.598403 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.600889 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.603720 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:10.625306 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:10.680004 7140 consensus_queue.cc:237] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:10.681528 9535 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:10.682042 9271 raft_consensus.cc:1261] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:10.682164 19369 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:10.683569 19369 consensus_queue.cc:1035] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:10.684150 17598 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:10.686208 9535 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:10.687060 9271 raft_consensus.cc:2932] T c80723cb0ce4446483a5f1754e066081 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:10.687707 6735 catalog_manager.cc:5484] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 6 to 7, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:10.693338 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet c80723cb0ce4446483a5f1754e066081 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58254
I20240501 07:59:10.693970 19775 tablet_replica.cc:330] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:59:10.694128 19775 raft_consensus.cc:2229] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.694339 19775 raft_consensus.cc:2258] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.694917 19775 ts_tablet_manager.cc:1875] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.697527 19775 ts_tablet_manager.cc:1888] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:10.697641 19775 log.cc:1196] T c80723cb0ce4446483a5f1754e066081 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c80723cb0ce4446483a5f1754e066081
I20240501 07:59:10.698191 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet c80723cb0ce4446483a5f1754e066081 (table rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]) successfully deleted
I20240501 07:59:10.847865 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.849678 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.855418 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:10.861425 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:10.863317 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.863587 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:10.867179 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:10.891660 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:10.950680 9534 consensus_queue.cc:237] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:10.952193 9266 raft_consensus.cc:1261] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:10.952744 6876 raft_consensus.cc:1261] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:10.952739 19534 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.953330 19534 consensus_queue.cc:1035] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:10.954701 19013 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:10.955477 6876 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:10.957592 9266 raft_consensus.cc:2932] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:10.959821 6736 catalog_manager.cc:5484] T 81e4af3cf8f044e2be0f8e3195f8c084 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:10.966133 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e4af3cf8f044e2be0f8e3195f8c084 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:10.966475 19847 tablet_replica.cc:330] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:10.966890 19847 raft_consensus.cc:2229] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.967056 19847 raft_consensus.cc:2258] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.967509 19847 ts_tablet_manager.cc:1875] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.970471 19847 ts_tablet_manager.cc:1888] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:10.970539 19847 log.cc:1196] T 81e4af3cf8f044e2be0f8e3195f8c084 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/81e4af3cf8f044e2be0f8e3195f8c084
I20240501 07:59:10.971149 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 81e4af3cf8f044e2be0f8e3195f8c084 (table rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]) successfully deleted
I20240501 07:59:11.110116 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.117578 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.119877 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.120155 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:11.128520 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.133386 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:11.134279 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.148113 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:11.232596 7010 consensus_queue.cc:237] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:11.233868 7140 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.233902 7271 raft_consensus.cc:1261] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.234299 19533 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.234596 17277 consensus_queue.cc:1035] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.236191 19533 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.236846 7140 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.237427 7270 raft_consensus.cc:2932] T 0c86818cabaf49678cfbcce3480f2645 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.240593 6735 catalog_manager.cc:5484] T 0c86818cabaf49678cfbcce3480f2645 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.244449 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c86818cabaf49678cfbcce3480f2645 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:11.244776 19847 tablet_replica.cc:330] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:11.245432 19847 raft_consensus.cc:2229] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.245628 19847 raft_consensus.cc:2258] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.246096 19847 ts_tablet_manager.cc:1875] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.248404 19847 ts_tablet_manager.cc:1888] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:11.248812 19847 log.cc:1196] T 0c86818cabaf49678cfbcce3480f2645 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0c86818cabaf49678cfbcce3480f2645
I20240501 07:59:11.249358 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 0c86818cabaf49678cfbcce3480f2645 (table rebalancer_extra_table_12 [id=8e7755a2edf143b9b232d3d0a15695ec]) successfully deleted
I20240501 07:59:11.406272 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.410295 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:11.422025 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.424423 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:11.425290 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.430604 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:11.433877 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.452685 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.513568 7140 consensus_queue.cc:237] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:11.514809 9665 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.515172 7010 raft_consensus.cc:1261] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.515254 19769 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.515651 19769 consensus_queue.cc:1035] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.517381 17598 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.521473 7010 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.521493 6733 catalog_manager.cc:5484] T c977cb9c5d1b41a29e7671218b917315 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.525769 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet c977cb9c5d1b41a29e7671218b917315 with delete_type TABLET_DATA_TOMBSTONED (TS 15ab96f6477b4291932409b80af07213 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58254
I20240501 07:59:11.527081 9665 raft_consensus.cc:2932] T c977cb9c5d1b41a29e7671218b917315 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 15ab96f6477b4291932409b80af07213 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.531209 19989 tablet_replica.cc:330] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: stopping tablet replica
I20240501 07:59:11.531409 19989 raft_consensus.cc:2229] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.531607 19989 raft_consensus.cc:2258] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.532238 19989 ts_tablet_manager.cc:1875] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.534719 19989 ts_tablet_manager.cc:1888] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:11.534783 19989 log.cc:1196] T c977cb9c5d1b41a29e7671218b917315 P 15ab96f6477b4291932409b80af07213: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c977cb9c5d1b41a29e7671218b917315
I20240501 07:59:11.535355 6722 catalog_manager.cc:4830] TS 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): tablet c977cb9c5d1b41a29e7671218b917315 (table rebalance_test_table_5 [id=c9f4bcc6262f4ecca1f57daaf60609df]) successfully deleted
I20240501 07:59:11.699996 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:11.706472 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.708128 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.715148 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:11.721854 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:11.726979 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.727422 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:11.762430 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:11.828680 7271 consensus_queue.cc:237] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:11.830019 9534 raft_consensus.cc:1261] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.830251 7140 raft_consensus.cc:1261] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.830396 18393 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.830583 18938 consensus_queue.cc:1035] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.832331 18938 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:11.832721 9534 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:11.833046 7140 raft_consensus.cc:2932] T 9051ba90bad3430fa2e128c88ec7b6f4 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:11.837280 6735 catalog_manager.cc:5484] T 9051ba90bad3430fa2e128c88ec7b6f4 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:11.839906 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 9051ba90bad3430fa2e128c88ec7b6f4 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:11.840303 20085 tablet_replica.cc:330] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:11.840468 20085 raft_consensus.cc:2229] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.840709 20085 raft_consensus.cc:2258] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.841327 20085 ts_tablet_manager.cc:1875] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.844635 20085 ts_tablet_manager.cc:1888] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:11.844708 20085 log.cc:1196] T 9051ba90bad3430fa2e128c88ec7b6f4 P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9051ba90bad3430fa2e128c88ec7b6f4
I20240501 07:59:11.845908 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 9051ba90bad3430fa2e128c88ec7b6f4 (table rebalance_test_table_1 [id=a1568fcce22f4efe9d2a8d214ec6ac0a]) successfully deleted
I20240501 07:59:11.874132 7270 consensus_queue.cc:237] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:11.875521 9266 raft_consensus.cc:1261] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.875558 9534 raft_consensus.cc:1261] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:11.875644 7206 consensus_peers.cc:485] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b683d8146a34fc6b74315f7db5d6890. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.875962 18393 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.876171 20112 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.877851 18393 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.878293 9266 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.880753 9534 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.881492 6737 catalog_manager.cc:5484] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:11.903409 9534 consensus_queue.cc:237] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:11.904840 9401 raft_consensus.cc:1261] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.904840 7139 raft_consensus.cc:1261] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:11.904877 9471 consensus_peers.cc:485] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80f08fbb906443759009170c6b5ad068. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.905313 19812 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.905514 20132 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.907661 20132 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:11.908236 7139 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:11.911530 6735 catalog_manager.cc:5484] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:11.912276 9401 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:11.934428 9665 consensus_queue.cc:237] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:11.936396 9534 raft_consensus.cc:1261] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.936470 9401 raft_consensus.cc:1261] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.936914 18564 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 07:59:11.939523 20151 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:11.939839 9601 consensus_peers.cc:485] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8123a9666b814a8eb93098bf850ebd72. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.943764 19127 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:11.944720 9535 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:11.945873 9401 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:11.947846 6737 catalog_manager.cc:5484] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:11.970546 9535 consensus_queue.cc:237] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:11.972069 9271 raft_consensus.cc:1261] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.972138 9401 raft_consensus.cc:1261] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:11.972294 9470 consensus_peers.cc:485] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7603404d6814a3ea8085a7d53772cb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.972605 20132 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.972798 19812 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.975179 20132 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.978379 9271 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.978935 6729 catalog_manager.cc:5484] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:11.979153 9401 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:12.002135 6876 consensus_queue.cc:237] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:12.003746 7010 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 07:59:12.003814 6815 consensus_peers.cc:485] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4882718ff90c4d939917f4970a9acb9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.003890 9271 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.004182 19024 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.004388 19748 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.006500 19024 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:12.006919 9271 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:12.008845 7010 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:12.010793 6735 catalog_manager.cc:5484] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 5 to 6, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:12.033260 9666 consensus_queue.cc:237] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.034714 9535 raft_consensus.cc:1261] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.034943 9401 raft_consensus.cc:1261] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:12.035115 9601 consensus_peers.cc:485] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b40ef631801d42ec954b19f7323dc00a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.035261 20151 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.035429 19127 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.037372 20151 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.037806 9535 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.039623 9401 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.042280 6737 catalog_manager.cc:5484] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.069350 9666 consensus_queue.cc:237] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:12.071070 9271 raft_consensus.cc:1261] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.071089 6876 raft_consensus.cc:1261] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:12.071359 9600 consensus_peers.cc:485] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7614555b22a847a2824f48d92611edda. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.071581 18564 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.071753 20151 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.073683 18564 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:12.074151 9271 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:12.075626 6876 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:12.077559 6729 catalog_manager.cc:5484] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:12.233276 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.236721 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:12.247839 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:12.251999 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.252763 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.253743 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:12.255077 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.280321 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.337132 20295 ts_tablet_manager.cc:902] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:12.337750 20295 tablet_copy_client.cc:308] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:12.338243 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet 4b683d8146a34fc6b74315f7db5d6890 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:49253)
I20240501 07:59:12.338364 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet 4b683d8146a34fc6b74315f7db5d6890 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:49253: session id = fc946354b5d04503a56122e59fc320be-4b683d8146a34fc6b74315f7db5d6890
I20240501 07:59:12.339298 7292 tablet_copy_source_session.cc:205] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.340252 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b683d8146a34fc6b74315f7db5d6890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.343072 20295 tablet_copy_client.cc:783] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.343250 20295 tablet_copy_client.cc:647] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.347095 20295 tablet_copy_client.cc:523] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.348496 20295 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:12.353293 20295 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.353536 20295 tablet_bootstrap.cc:492] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:12.353679 20295 ts_tablet_manager.cc:1367] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:12.354115 20295 raft_consensus.cc:348] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:12.354240 20295 raft_consensus.cc:729] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.354377 20295 consensus_queue.cc:260] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:12.355499 20295 ts_tablet_manager.cc:1398] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:12.356151 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-4b683d8146a34fc6b74315f7db5d6890 received from {username='slave'} at 127.0.99.131:49253
I20240501 07:59:12.356235 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session fc946354b5d04503a56122e59fc320be-4b683d8146a34fc6b74315f7db5d6890 on tablet 4b683d8146a34fc6b74315f7db5d6890 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:12.363088 20301 ts_tablet_manager.cc:902] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:12.363688 20301 tablet_copy_client.cc:308] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:12.364152 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 80f08fbb906443759009170c6b5ad068 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:54625)
I20240501 07:59:12.364267 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 80f08fbb906443759009170c6b5ad068 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:54625: session id = 9a3888360d4d4ace96b6b909c160bfc2-80f08fbb906443759009170c6b5ad068
I20240501 07:59:12.365269 9555 tablet_copy_source_session.cc:205] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.366242 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f08fbb906443759009170c6b5ad068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.368886 20301 tablet_copy_client.cc:783] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.369040 20301 tablet_copy_client.cc:647] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.370702 20301 tablet_copy_client.cc:523] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.371968 20301 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:12.376600 20301 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.376878 20301 tablet_bootstrap.cc:492] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:12.377049 20301 ts_tablet_manager.cc:1367] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 07:59:12.377732 20301 raft_consensus.cc:348] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:12.377849 20301 raft_consensus.cc:729] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.377974 20301 consensus_queue.cc:260] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:12.378475 20301 ts_tablet_manager.cc:1398] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:12.379179 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-80f08fbb906443759009170c6b5ad068 received from {username='slave'} at 127.0.99.130:54625
I20240501 07:59:12.379263 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-80f08fbb906443759009170c6b5ad068 on tablet 80f08fbb906443759009170c6b5ad068 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:12.441702 20337 ts_tablet_manager.cc:902] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:12.442282 20337 tablet_copy_client.cc:308] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:12.442814 9685 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 8123a9666b814a8eb93098bf850ebd72 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:41375)
I20240501 07:59:12.442922 9685 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 8123a9666b814a8eb93098bf850ebd72 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:41375: session id = 1a9029bee6a144938f68b363e4cca162-8123a9666b814a8eb93098bf850ebd72
I20240501 07:59:12.443960 9685 tablet_copy_source_session.cc:205] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.445163 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8123a9666b814a8eb93098bf850ebd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.448518 20337 tablet_copy_client.cc:783] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.448788 20337 tablet_copy_client.cc:647] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.452358 20337 tablet_copy_client.cc:523] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.453725 20337 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:12.458577 20337 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.458917 20337 tablet_bootstrap.cc:492] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:12.459241 20337 ts_tablet_manager.cc:1367] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:12.459821 20337 raft_consensus.cc:348] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.459967 20337 raft_consensus.cc:729] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.460155 20337 consensus_queue.cc:260] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.460554 20337 ts_tablet_manager.cc:1398] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.461310 9685 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-8123a9666b814a8eb93098bf850ebd72 received from {username='slave'} at 127.0.99.129:41375
I20240501 07:59:12.461403 9685 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-8123a9666b814a8eb93098bf850ebd72 on tablet 8123a9666b814a8eb93098bf850ebd72 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:12.520467 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.521613 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:12.522248 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.524868 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.525292 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.531236 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:12.558504 20295 ts_tablet_manager.cc:902] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:12.559613 20295 tablet_copy_client.cc:308] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:12.560643 9685 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet 7614555b22a847a2824f48d92611edda from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:40629)
I20240501 07:59:12.560883 9685 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet 7614555b22a847a2824f48d92611edda from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:40629: session id = fc946354b5d04503a56122e59fc320be-7614555b22a847a2824f48d92611edda
I20240501 07:59:12.562924 9685 tablet_copy_source_session.cc:205] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.564020 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7614555b22a847a2824f48d92611edda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.566238 20295 tablet_copy_client.cc:783] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.566395 20295 tablet_copy_client.cc:647] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.568118 20295 tablet_copy_client.cc:523] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.569653 20295 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:12.577543 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:12.587945 20359 ts_tablet_manager.cc:902] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:12.588604 20359 tablet_copy_client.cc:308] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37209
I20240501 07:59:12.589345 6899 tablet_copy_service.cc:140] P 1a9029bee6a144938f68b363e4cca162: Received BeginTabletCopySession request for tablet 4882718ff90c4d939917f4970a9acb9e from peer ac6c8bc1b74b483791e262abee7862cf ({username='slave'} at 127.0.99.135:34317)
I20240501 07:59:12.589457 6899 tablet_copy_service.cc:161] P 1a9029bee6a144938f68b363e4cca162: Beginning new tablet copy session on tablet 4882718ff90c4d939917f4970a9acb9e from peer ac6c8bc1b74b483791e262abee7862cf at {username='slave'} at 127.0.99.135:34317: session id = ac6c8bc1b74b483791e262abee7862cf-4882718ff90c4d939917f4970a9acb9e
I20240501 07:59:12.590551 6899 tablet_copy_source_session.cc:205] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.595685 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4882718ff90c4d939917f4970a9acb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.595870 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.598258 20359 tablet_copy_client.cc:783] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.598424 20359 tablet_copy_client.cc:647] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.599184 20337 ts_tablet_manager.cc:902] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:12.595949 20364 ts_tablet_manager.cc:902] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:12.599642 20337 tablet_copy_client.cc:308] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:35111
I20240501 07:59:12.600045 20364 tablet_copy_client.cc:308] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:12.600103 9685 tablet_copy_service.cc:140] P 7fa3a88c25964445975bb47e0feed460: Received BeginTabletCopySession request for tablet b40ef631801d42ec954b19f7323dc00a from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:41375)
I20240501 07:59:12.600204 9685 tablet_copy_service.cc:161] P 7fa3a88c25964445975bb47e0feed460: Beginning new tablet copy session on tablet b40ef631801d42ec954b19f7323dc00a from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:41375: session id = 1a9029bee6a144938f68b363e4cca162-b40ef631801d42ec954b19f7323dc00a
I20240501 07:59:12.600473 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet a7603404d6814a3ea8085a7d53772cb7 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:48773)
I20240501 07:59:12.600564 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet a7603404d6814a3ea8085a7d53772cb7 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:48773: session id = 7fa3a88c25964445975bb47e0feed460-a7603404d6814a3ea8085a7d53772cb7
I20240501 07:59:12.601387 9685 tablet_copy_source_session.cc:205] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.601557 9555 tablet_copy_source_session.cc:205] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.603771 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40ef631801d42ec954b19f7323dc00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.606015 20337 tablet_copy_client.cc:783] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.606156 20337 tablet_copy_client.cc:647] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.608119 20359 tablet_copy_client.cc:523] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.608691 20337 tablet_copy_client.cc:523] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.609189 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7603404d6814a3ea8085a7d53772cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.609520 20359 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Bootstrap starting.
I20240501 07:59:12.609988 20337 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:12.615792 20364 tablet_copy_client.cc:783] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.616007 20364 tablet_copy_client.cc:647] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.617120 20364 tablet_copy_client.cc:523] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.620327 20364 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:12.620705 20337 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.621047 20337 tablet_bootstrap.cc:492] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:12.621212 20337 ts_tablet_manager.cc:1367] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 07:59:12.621872 20337 raft_consensus.cc:348] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.622074 20337 raft_consensus.cc:729] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.622217 20337 consensus_queue.cc:260] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.622531 20337 ts_tablet_manager.cc:1398] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.622978 9685 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-b40ef631801d42ec954b19f7323dc00a received from {username='slave'} at 127.0.99.129:41375
I20240501 07:59:12.623062 9685 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-b40ef631801d42ec954b19f7323dc00a on tablet b40ef631801d42ec954b19f7323dc00a with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:12.623642 20295 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.623914 20295 tablet_bootstrap.cc:492] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:12.624114 20295 ts_tablet_manager.cc:1367] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.003s
I20240501 07:59:12.624590 20295 raft_consensus.cc:348] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:12.624711 20295 raft_consensus.cc:729] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.624866 20295 consensus_queue.cc:260] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:12.625180 20295 ts_tablet_manager.cc:1398] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:12.632494 9685 tablet_copy_service.cc:342] P 7fa3a88c25964445975bb47e0feed460: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-7614555b22a847a2824f48d92611edda received from {username='slave'} at 127.0.99.131:40629
I20240501 07:59:12.632586 9685 tablet_copy_service.cc:433] P 7fa3a88c25964445975bb47e0feed460: ending tablet copy session fc946354b5d04503a56122e59fc320be-7614555b22a847a2824f48d92611edda on tablet 7614555b22a847a2824f48d92611edda with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:12.642135 20364 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.642405 20364 tablet_bootstrap.cc:492] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:59:12.642568 20364 ts_tablet_manager.cc:1367] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 07:59:12.644649 20364 raft_consensus.cc:348] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:12.644778 20364 raft_consensus.cc:729] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.644995 20364 consensus_queue.cc:260] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:12.646641 20364 ts_tablet_manager.cc:1398] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:12.647111 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-a7603404d6814a3ea8085a7d53772cb7 received from {username='slave'} at 127.0.99.136:48773
I20240501 07:59:12.647208 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-a7603404d6814a3ea8085a7d53772cb7 on tablet a7603404d6814a3ea8085a7d53772cb7 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:12.662348 20359 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.662667 20359 tablet_bootstrap.cc:492] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Bootstrap complete.
I20240501 07:59:12.662921 20359 ts_tablet_manager.cc:1367] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Time spent bootstrapping tablet: real 0.053s user 0.029s sys 0.000s
I20240501 07:59:12.663412 20359 raft_consensus.cc:348] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:12.663513 20359 raft_consensus.cc:729] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.663637 20359 consensus_queue.cc:260] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:12.663921 20359 ts_tablet_manager.cc:1398] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:12.664580 6899 tablet_copy_service.cc:342] P 1a9029bee6a144938f68b363e4cca162: Request end of tablet copy session ac6c8bc1b74b483791e262abee7862cf-4882718ff90c4d939917f4970a9acb9e received from {username='slave'} at 127.0.99.135:34317
I20240501 07:59:12.664682 6899 tablet_copy_service.cc:433] P 1a9029bee6a144938f68b363e4cca162: ending tablet copy session ac6c8bc1b74b483791e262abee7862cf-4882718ff90c4d939917f4970a9acb9e on tablet 4882718ff90c4d939917f4970a9acb9e with peer ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:12.728298 7140 raft_consensus.cc:1203] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:12.805940 7009 raft_consensus.cc:1203] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:12.818881 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.822355 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:12.823261 6876 raft_consensus.cc:1203] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:12.827900 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:12.828178 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.837601 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:12.839309 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.843114 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:12.860265 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:12.933477 9666 consensus_queue.cc:237] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:12.934727 6878 raft_consensus.cc:1261] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:12.934711 9401 raft_consensus.cc:1261] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:12.935168 19127 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:12.935312 20431 consensus_queue.cc:1035] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:12.936848 19127 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.937206 9401 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.937399 6878 raft_consensus.cc:2932] T 8123a9666b814a8eb93098bf850ebd72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.940918 6731 catalog_manager.cc:5484] T 8123a9666b814a8eb93098bf850ebd72 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:12.943953 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 8123a9666b814a8eb93098bf850ebd72 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:12.944344 20455 tablet_replica.cc:330] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:12.944561 20455 raft_consensus.cc:2229] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.944761 20455 raft_consensus.cc:2258] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.946270 20455 ts_tablet_manager.cc:1875] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.948683 20455 ts_tablet_manager.cc:1888] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:12.948786 20455 log.cc:1196] T 8123a9666b814a8eb93098bf850ebd72 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/8123a9666b814a8eb93098bf850ebd72
I20240501 07:59:12.949308 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 8123a9666b814a8eb93098bf850ebd72 (table rebalancer_extra_table_34 [id=5131e3bfcbbe4c7197960cf482ff02c9]) successfully deleted
I20240501 07:59:12.964391 9534 raft_consensus.cc:1203] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:13.083348 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.089742 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.089838 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:13.092404 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.099174 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.102953 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:13.105381 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.115543 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:13.121002 7139 raft_consensus.cc:1203] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:13.159236 6878 raft_consensus.cc:1203] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:13.173148 9666 raft_consensus.cc:1203] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:13.194932 9535 consensus_queue.cc:237] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:13.196406 7139 raft_consensus.cc:1261] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.196388 7010 raft_consensus.cc:1261] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.196905 19013 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.197031 20419 consensus_queue.cc:1035] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.199088 19013 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:13.199451 7010 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:13.203545 6730 catalog_manager.cc:5484] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:13.204437 7139 raft_consensus.cc:2932] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:13.206801 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 80f08fbb906443759009170c6b5ad068 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:13.209065 20523 tablet_replica.cc:330] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:13.209268 20523 raft_consensus.cc:2229] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.209445 20523 raft_consensus.cc:2258] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.210202 20523 ts_tablet_manager.cc:1875] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.212707 20523 ts_tablet_manager.cc:1888] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:13.212821 20523 log.cc:1196] T 80f08fbb906443759009170c6b5ad068 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/80f08fbb906443759009170c6b5ad068
I20240501 07:59:13.213443 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 80f08fbb906443759009170c6b5ad068 (table rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]) successfully deleted
I20240501 07:59:13.376437 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.378690 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:13.382436 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.385921 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.388868 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:13.389133 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.401964 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:13.449209 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.494036 7270 consensus_queue.cc:237] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:13.495334 9535 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:13.496475 9666 raft_consensus.cc:1261] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:13.496901 18393 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:13.497138 20115 consensus_queue.cc:1035] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:13.499095 20594 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:13.499456 9666 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:13.499456 9535 raft_consensus.cc:2932] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:13.505950 6731 catalog_manager.cc:5484] T ab7dcdd38eff43fe9fff9081ee644c8d P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:13.508553 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7dcdd38eff43fe9fff9081ee644c8d with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:13.508939 20600 tablet_replica.cc:330] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:13.509105 20600 raft_consensus.cc:2229] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.509289 20600 raft_consensus.cc:2258] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.509842 20600 ts_tablet_manager.cc:1875] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.512356 20600 ts_tablet_manager.cc:1888] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:13.512459 20600 log.cc:1196] T ab7dcdd38eff43fe9fff9081ee644c8d P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ab7dcdd38eff43fe9fff9081ee644c8d
I20240501 07:59:13.515153 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet ab7dcdd38eff43fe9fff9081ee644c8d (table rebalance_test_table_17 [id=6cb6a410f06449a1a9bb80e4542585f2]) successfully deleted
I20240501 07:59:13.656399 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.664978 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:13.667641 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.667680 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.670145 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.670802 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:13.679029 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:13.688237 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.768867 7270 consensus_queue.cc:237] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:13.770069 9535 raft_consensus.cc:1261] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.770231 7139 raft_consensus.cc:1261] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.770494 20594 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.770658 18393 consensus_queue.cc:1035] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.772403 20115 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:13.772786 7139 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:13.772935 9535 raft_consensus.cc:2932] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:13.776528 6734 catalog_manager.cc:5484] T 4b683d8146a34fc6b74315f7db5d6890 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:13.780009 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b683d8146a34fc6b74315f7db5d6890 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:13.780458 20667 tablet_replica.cc:330] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:13.780669 20667 raft_consensus.cc:2229] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.780829 20667 raft_consensus.cc:2258] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.781317 20667 ts_tablet_manager.cc:1875] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.784407 20667 ts_tablet_manager.cc:1888] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:13.784525 20667 log.cc:1196] T 4b683d8146a34fc6b74315f7db5d6890 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4b683d8146a34fc6b74315f7db5d6890
I20240501 07:59:13.785094 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 4b683d8146a34fc6b74315f7db5d6890 (table rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]) successfully deleted
I20240501 07:59:13.923667 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.931082 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:13.941390 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.942571 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:13.943158 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:13.955866 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:13.956282 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:13.959604 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.037560 9266 consensus_queue.cc:237] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:14.038731 9401 raft_consensus.cc:1261] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.038750 7270 raft_consensus.cc:1261] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.039152 19111 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.039341 20680 consensus_queue.cc:1035] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.040961 20679 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:14.041520 7270 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:14.041532 9401 raft_consensus.cc:2932] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:14.044701 6728 catalog_manager.cc:5484] T f7c14004fdb24e96bacbf8589518f3b1 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:14.048182 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c14004fdb24e96bacbf8589518f3b1 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:14.048540 20740 tablet_replica.cc:330] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:14.048724 20740 raft_consensus.cc:2229] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.048877 20740 raft_consensus.cc:2258] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.049386 20740 ts_tablet_manager.cc:1875] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.052132 20740 ts_tablet_manager.cc:1888] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:14.052297 20740 log.cc:1196] T f7c14004fdb24e96bacbf8589518f3b1 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f7c14004fdb24e96bacbf8589518f3b1
I20240501 07:59:14.052937 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet f7c14004fdb24e96bacbf8589518f3b1 (table rebalancer_extra_table_32 [id=bb2f999cca1d4027a9ffb33888b8f913]) successfully deleted
I20240501 07:59:14.192552 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.199391 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.204622 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:14.205195 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:14.214200 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.245221 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:14.269490 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.269932 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.318887 7271 consensus_queue.cc:237] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:14.320232 9666 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:14.320295 9401 raft_consensus.cc:1261] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:14.320904 20115 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.321128 18393 consensus_queue.cc:1035] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.323287 18393 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:14.325850 9666 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:14.326041 9403 raft_consensus.cc:2932] T 4e7f613e16c245de83fde8c277b8d350 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:14.329260 6731 catalog_manager.cc:5484] T 4e7f613e16c245de83fde8c277b8d350 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 6 to 7, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:14.332573 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7f613e16c245de83fde8c277b8d350 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:14.334287 20812 tablet_replica.cc:330] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:14.334510 20812 raft_consensus.cc:2229] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.334792 20812 raft_consensus.cc:2258] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.335491 20812 ts_tablet_manager.cc:1875] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.337735 20812 ts_tablet_manager.cc:1888] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:14.337867 20812 log.cc:1196] T 4e7f613e16c245de83fde8c277b8d350 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e7f613e16c245de83fde8c277b8d350
I20240501 07:59:14.338642 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 4e7f613e16c245de83fde8c277b8d350 (table rebalance_test_table_11 [id=e3f92d832c844583ae778357ccc93086]) successfully deleted
I20240501 07:59:14.489743 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:14.492560 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.494302 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.502617 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:14.508980 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:14.522332 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.524818 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.548175 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.617674 6878 consensus_queue.cc:237] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:14.619161 9401 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:14.619210 9266 raft_consensus.cc:1261] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:14.619657 20522 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.619889 20886 consensus_queue.cc:1035] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.621853 20522 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:14.622249 9266 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:14.625290 9401 raft_consensus.cc:2932] T d30d4d6ddf3a40989b10cf5c10ef7082 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:14.625489 6734 catalog_manager.cc:5484] T d30d4d6ddf3a40989b10cf5c10ef7082 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:14.628859 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet d30d4d6ddf3a40989b10cf5c10ef7082 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:14.629215 20889 tablet_replica.cc:330] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:14.631286 20889 raft_consensus.cc:2229] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.631502 20889 raft_consensus.cc:2258] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.632093 20889 ts_tablet_manager.cc:1875] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.634670 20889 ts_tablet_manager.cc:1888] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 07:59:14.634800 20889 log.cc:1196] T d30d4d6ddf3a40989b10cf5c10ef7082 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d30d4d6ddf3a40989b10cf5c10ef7082
I20240501 07:59:14.635391 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet d30d4d6ddf3a40989b10cf5c10ef7082 (table rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]) successfully deleted
I20240501 07:59:14.789407 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:14.791414 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:14.791988 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.796387 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.798677 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:14.798765 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:14.819146 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.823308 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:14.884503 9666 consensus_queue.cc:237] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:14.885748 7139 raft_consensus.cc:1261] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.886130 6876 raft_consensus.cc:1261] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.886365 20947 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.886547 20948 consensus_queue.cc:1035] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.888573 20648 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:14.889015 6876 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:14.892501 6737 catalog_manager.cc:5484] T 7614555b22a847a2824f48d92611edda P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:14.894268 7139 raft_consensus.cc:2932] T 7614555b22a847a2824f48d92611edda P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:14.895265 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 7614555b22a847a2824f48d92611edda with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:14.895622 20972 tablet_replica.cc:330] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:14.895787 20972 raft_consensus.cc:2229] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.896041 20972 raft_consensus.cc:2258] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.896540 20972 ts_tablet_manager.cc:1875] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.899827 20972 ts_tablet_manager.cc:1888] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:14.899900 20972 log.cc:1196] T 7614555b22a847a2824f48d92611edda P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7614555b22a847a2824f48d92611edda
I20240501 07:59:14.900457 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 7614555b22a847a2824f48d92611edda (table rebalancer_extra_table_20 [id=fd084607dede46f48a7d30d818048b63]) successfully deleted
I20240501 07:59:15.068373 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.075922 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.081068 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:15.081712 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.083992 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:15.086072 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.092211 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:15.099562 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.181459 9534 consensus_queue.cc:237] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:15.182741 9665 raft_consensus.cc:1261] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.182794 9401 raft_consensus.cc:1261] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.183080 20419 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.183264 19013 consensus_queue.cc:1035] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.185104 20419 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.186878 9401 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.187072 9665 raft_consensus.cc:2932] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.191118 6731 catalog_manager.cc:5484] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.194463 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet a7603404d6814a3ea8085a7d53772cb7 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:15.194679 20972 tablet_replica.cc:330] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:15.194806 20972 raft_consensus.cc:2229] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.194911 20972 raft_consensus.cc:2258] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.195389 20972 ts_tablet_manager.cc:1875] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.197891 20972 ts_tablet_manager.cc:1888] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:15.197974 20972 log.cc:1196] T a7603404d6814a3ea8085a7d53772cb7 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7603404d6814a3ea8085a7d53772cb7
I20240501 07:59:15.198521 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet a7603404d6814a3ea8085a7d53772cb7 (table rebalancer_extra_table_18 [id=f2bef1cb187c4728b3888d496f787cf8]) successfully deleted
I20240501 07:59:15.223855 9534 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.225520 9665 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:15.225625 9266 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:15.226001 20419 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:15.226302 20419 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:15.226644 9471 consensus_peers.cc:485] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee29c58cdc6544e8933300ad5faecb3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.230060 20419 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.232156 9665 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.233711 9266 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.235821 6737 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 4 to 5, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.258618 7010 consensus_queue.cc:237] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
W20240501 07:59:15.260252 6945 consensus_peers.cc:485] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a54e29c093e4be485bf1af1e9ea7a72. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.260279 9266 raft_consensus.cc:1261] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.260680 21100 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.260783 6876 raft_consensus.cc:1261] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.261269 21100 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.263155 19965 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.263643 6876 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.267362 9266 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.269866 6733 catalog_manager.cc:5484] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index -1 to 2, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:15.291692 9401 consensus_queue.cc:237] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:15.293226 9266 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:15.293256 9338 consensus_peers.cc:485] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b78be08b353141469d49c7460393a051. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.293323 7010 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.293586 20978 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.293782 18254 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.295734 20978 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:15.296119 7010 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:15.298622 9266 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:15.299582 6737 catalog_manager.cc:5484] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 1a9029bee6a144938f68b363e4cca162 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:15.320782 9401 consensus_queue.cc:237] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.322937 9665 raft_consensus.cc:1261] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:15.323181 9339 consensus_peers.cc:485] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b09efd2448a4be7b63636b66e56a030. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.323411 21122 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.323536 9534 raft_consensus.cc:1261] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.323872 21122 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.326643 20792 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.328351 9534 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.330713 9665 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.334693 6734 catalog_manager.cc:5484] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.363517 9534 consensus_queue.cc:237] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:59:15.365085 9469 consensus_peers.cc:485] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aea3c6384b74a32a7605f9777534618. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.365281 9271 raft_consensus.cc:1261] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.365703 9401 raft_consensus.cc:1261] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.366035 19013 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.366333 21083 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.368517 19013 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:15.368933 9401 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:15.371661 9271 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:15.372357 6731 catalog_manager.cc:5484] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:15.397305 9534 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
W20240501 07:59:15.398883 9471 consensus_peers.cc:485] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8921a8351e04daba09a8e7e4f683fd9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.399075 7139 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.399168 9666 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.399472 20419 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.399828 19013 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.402084 19013 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.403954 9666 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.404839 7139 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.405652 6737 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 3 to 4, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:15.556327 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:15.556768 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.556825 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:15.561657 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.575174 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.584863 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:15.590634 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.602942 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.655151 21241 ts_tablet_manager.cc:902] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:15.655689 21241 tablet_copy_client.cc:308] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:15.657346 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet ee29c58cdc6544e8933300ad5faecb3a from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:54625)
I20240501 07:59:15.657464 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet ee29c58cdc6544e8933300ad5faecb3a from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:54625: session id = 9a3888360d4d4ace96b6b909c160bfc2-ee29c58cdc6544e8933300ad5faecb3a
I20240501 07:59:15.658213 9555 tablet_copy_source_session.cc:205] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.659063 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee29c58cdc6544e8933300ad5faecb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.662207 21241 tablet_copy_client.cc:783] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.662431 21241 tablet_copy_client.cc:647] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.667119 21241 tablet_copy_client.cc:523] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.668525 21241 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:15.673532 21241 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.673802 21241 tablet_bootstrap.cc:492] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:15.673967 21241 ts_tablet_manager.cc:1367] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 07:59:15.674458 21241 raft_consensus.cc:348] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.674594 21241 raft_consensus.cc:729] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.674733 21241 consensus_queue.cc:260] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.675159 21241 ts_tablet_manager.cc:1398] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 07:59:15.676970 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-ee29c58cdc6544e8933300ad5faecb3a received from {username='slave'} at 127.0.99.130:54625
I20240501 07:59:15.677052 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-ee29c58cdc6544e8933300ad5faecb3a on tablet ee29c58cdc6544e8933300ad5faecb3a with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:15.761780 21278 ts_tablet_manager.cc:902] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:15.762645 21279 ts_tablet_manager.cc:902] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:15.763211 21279 tablet_copy_client.cc:308] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:59:15.763351 21278 tablet_copy_client.cc:308] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39675
I20240501 07:59:15.764642 7030 tablet_copy_service.cc:140] P 9a3888360d4d4ace96b6b909c160bfc2: Received BeginTabletCopySession request for tablet 3a54e29c093e4be485bf1af1e9ea7a72 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:39427)
I20240501 07:59:15.764753 7030 tablet_copy_service.cc:161] P 9a3888360d4d4ace96b6b909c160bfc2: Beginning new tablet copy session on tablet 3a54e29c093e4be485bf1af1e9ea7a72 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:39427: session id = 7fa3a88c25964445975bb47e0feed460-3a54e29c093e4be485bf1af1e9ea7a72
I20240501 07:59:15.765511 7030 tablet_copy_source_session.cc:205] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.766412 21278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a54e29c093e4be485bf1af1e9ea7a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.766446 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet b78be08b353141469d49c7460393a051 from peer 1a9029bee6a144938f68b363e4cca162 ({username='slave'} at 127.0.99.129:34851)
I20240501 07:59:15.766539 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet b78be08b353141469d49c7460393a051 from peer 1a9029bee6a144938f68b363e4cca162 at {username='slave'} at 127.0.99.129:34851: session id = 1a9029bee6a144938f68b363e4cca162-b78be08b353141469d49c7460393a051
I20240501 07:59:15.767544 9414 tablet_copy_source_session.cc:205] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.769304 21279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78be08b353141469d49c7460393a051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.770385 21278 tablet_copy_client.cc:783] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.770570 21278 tablet_copy_client.cc:647] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.772123 21279 tablet_copy_client.cc:783] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.772346 21279 tablet_copy_client.cc:647] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.774520 21278 tablet_copy_client.cc:523] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.776810 21278 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:15.784781 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.785269 21279 tablet_copy_client.cc:523] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.786826 21279 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Bootstrap starting.
I20240501 07:59:15.790271 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.796170 21278 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.796458 21278 tablet_bootstrap.cc:492] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:59:15.796627 21278 ts_tablet_manager.cc:1367] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240501 07:59:15.797226 21278 raft_consensus.cc:348] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:15.797345 21278 raft_consensus.cc:729] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.797456 21278 consensus_queue.cc:260] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:15.797838 21278 ts_tablet_manager.cc:1398] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.798332 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:15.798476 7030 tablet_copy_service.cc:342] P 9a3888360d4d4ace96b6b909c160bfc2: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-3a54e29c093e4be485bf1af1e9ea7a72 received from {username='slave'} at 127.0.99.136:39427
I20240501 07:59:15.798548 7030 tablet_copy_service.cc:433] P 9a3888360d4d4ace96b6b909c160bfc2: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-3a54e29c093e4be485bf1af1e9ea7a72 on tablet 3a54e29c093e4be485bf1af1e9ea7a72 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:15.804454 21279 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.804742 21279 tablet_bootstrap.cc:492] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Bootstrap complete.
I20240501 07:59:15.804931 21279 ts_tablet_manager.cc:1367] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20240501 07:59:15.805440 21279 raft_consensus.cc:348] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:15.805564 21279 raft_consensus.cc:729] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.805691 21279 consensus_queue.cc:260] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:15.806674 21279 ts_tablet_manager.cc:1398] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:15.810827 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 1a9029bee6a144938f68b363e4cca162-b78be08b353141469d49c7460393a051 received from {username='slave'} at 127.0.99.129:34851
I20240501 07:59:15.810905 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 1a9029bee6a144938f68b363e4cca162-b78be08b353141469d49c7460393a051 on tablet b78be08b353141469d49c7460393a051 with peer 1a9029bee6a144938f68b363e4cca162
I20240501 07:59:15.832458 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.838630 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:15.840152 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:15.851984 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:15.855186 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:15.892580 21241 ts_tablet_manager.cc:902] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:15.893054 21241 tablet_copy_client.cc:308] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:59:15.893783 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet 2b09efd2448a4be7b63636b66e56a030 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:53171)
I20240501 07:59:15.893899 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet 2b09efd2448a4be7b63636b66e56a030 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:53171: session id = 9a3888360d4d4ace96b6b909c160bfc2-2b09efd2448a4be7b63636b66e56a030
I20240501 07:59:15.894707 9414 tablet_copy_source_session.cc:205] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.895632 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b09efd2448a4be7b63636b66e56a030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.898010 21241 tablet_copy_client.cc:783] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.898151 21241 tablet_copy_client.cc:647] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.900089 21241 tablet_copy_client.cc:523] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.901463 21241 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:15.907020 21241 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.907280 21241 tablet_bootstrap.cc:492] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:15.907461 21241 ts_tablet_manager.cc:1367] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 07:59:15.907965 21241 raft_consensus.cc:348] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.908084 21241 raft_consensus.cc:729] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.908216 21241 consensus_queue.cc:260] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:15.908494 21241 ts_tablet_manager.cc:1398] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.908960 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-2b09efd2448a4be7b63636b66e56a030 received from {username='slave'} at 127.0.99.130:53171
I20240501 07:59:15.909039 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-2b09efd2448a4be7b63636b66e56a030 on tablet 2b09efd2448a4be7b63636b66e56a030 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:15.929765 21323 ts_tablet_manager.cc:902] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:15.930469 21323 tablet_copy_client.cc:308] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:15.930917 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 4aea3c6384b74a32a7605f9777534618 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:43037)
I20240501 07:59:15.931118 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 4aea3c6384b74a32a7605f9777534618 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:43037: session id = fc946354b5d04503a56122e59fc320be-4aea3c6384b74a32a7605f9777534618
I20240501 07:59:15.932163 9555 tablet_copy_source_session.cc:205] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.933048 21323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aea3c6384b74a32a7605f9777534618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.935290 21323 tablet_copy_client.cc:783] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.935446 21323 tablet_copy_client.cc:647] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.936517 21323 tablet_copy_client.cc:523] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.938102 21323 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:15.943079 21323 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.943413 21323 tablet_bootstrap.cc:492] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:15.943596 21323 ts_tablet_manager.cc:1367] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 07:59:15.944034 21323 raft_consensus.cc:348] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:15.944136 21323 raft_consensus.cc:729] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.944269 21323 consensus_queue.cc:260] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:15.946116 21323 ts_tablet_manager.cc:1398] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:15.947088 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-4aea3c6384b74a32a7605f9777534618 received from {username='slave'} at 127.0.99.131:43037
I20240501 07:59:15.947167 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fc946354b5d04503a56122e59fc320be-4aea3c6384b74a32a7605f9777534618 on tablet 4aea3c6384b74a32a7605f9777534618 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:16.023758 21241 ts_tablet_manager.cc:902] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:16.024260 21241 tablet_copy_client.cc:308] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:16.024928 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet f8921a8351e04daba09a8e7e4f683fd9 from peer 9a3888360d4d4ace96b6b909c160bfc2 ({username='slave'} at 127.0.99.130:54625)
I20240501 07:59:16.025033 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet f8921a8351e04daba09a8e7e4f683fd9 from peer 9a3888360d4d4ace96b6b909c160bfc2 at {username='slave'} at 127.0.99.130:54625: session id = 9a3888360d4d4ace96b6b909c160bfc2-f8921a8351e04daba09a8e7e4f683fd9
I20240501 07:59:16.026044 9555 tablet_copy_source_session.cc:205] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:16.027205 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8921a8351e04daba09a8e7e4f683fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.029453 21241 tablet_copy_client.cc:783] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:16.029611 21241 tablet_copy_client.cc:647] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:16.031294 21241 tablet_copy_client.cc:523] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:16.032367 21241 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap starting.
I20240501 07:59:16.032660 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.042258 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:16.047173 21241 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:16.047463 21241 tablet_bootstrap.cc:492] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Bootstrap complete.
I20240501 07:59:16.047652 21241 ts_tablet_manager.cc:1367] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240501 07:59:16.048454 21241 raft_consensus.cc:348] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:16.048602 21241 raft_consensus.cc:729] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.048745 21241 consensus_queue.cc:260] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:16.050124 21241 ts_tablet_manager.cc:1398] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:16.050592 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-f8921a8351e04daba09a8e7e4f683fd9 received from {username='slave'} at 127.0.99.130:54625
I20240501 07:59:16.050691 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session 9a3888360d4d4ace96b6b909c160bfc2-f8921a8351e04daba09a8e7e4f683fd9 on tablet f8921a8351e04daba09a8e7e4f683fd9 with peer 9a3888360d4d4ace96b6b909c160bfc2
I20240501 07:59:16.051313 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:16.058808 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.059134 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.081496 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.098556 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:16.116927 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.221529 7009 raft_consensus.cc:1203] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240501 07:59:16.293900 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.299386 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.306397 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:16.307272 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.310672 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:16.310658 6877 raft_consensus.cc:1203] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:16.311699 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:16.322988 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.343223 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.363615 9666 raft_consensus.cc:1203] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:16.425925 7009 raft_consensus.cc:1203] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:16.495415 7009 raft_consensus.cc:1203] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:16.506137 7140 raft_consensus.cc:1203] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:16.551777 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.562911 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:16.567205 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.571273 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:16.571434 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:16.573567 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.597939 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.616169 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.664619 9534 consensus_queue.cc:237] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:16.665904 7140 raft_consensus.cc:1261] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.665936 9403 raft_consensus.cc:1261] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.666293 20419 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.666445 21057 consensus_queue.cc:1035] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.668140 21057 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:16.668602 9403 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:16.668787 7140 raft_consensus.cc:2932] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:16.671938 6733 catalog_manager.cc:5484] T 4aea3c6384b74a32a7605f9777534618 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:16.676229 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aea3c6384b74a32a7605f9777534618 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:16.676563 21497 tablet_replica.cc:330] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:16.676748 21497 raft_consensus.cc:2229] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.676891 21497 raft_consensus.cc:2258] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.677443 21497 ts_tablet_manager.cc:1875] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.679759 21497 ts_tablet_manager.cc:1888] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:16.679831 21497 log.cc:1196] T 4aea3c6384b74a32a7605f9777534618 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4aea3c6384b74a32a7605f9777534618
I20240501 07:59:16.680502 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 4aea3c6384b74a32a7605f9777534618 (table rebalancer_extra_table_28 [id=43ba3c91d89e4d8f84c14a8ba9138816]) successfully deleted
I20240501 07:59:16.814730 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.823519 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:16.824146 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:16.835605 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.835646 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:16.852080 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.856886 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:16.875990 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:16.926612 7009 consensus_queue.cc:237] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:16.927789 9665 raft_consensus.cc:1261] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.927798 6877 raft_consensus.cc:1261] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.928141 21103 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.928369 17277 consensus_queue.cc:1035] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.929797 21103 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:16.930305 6877 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:16.931939 9666 raft_consensus.cc:2932] T 3a54e29c093e4be485bf1af1e9ea7a72 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:16.935200 6731 catalog_manager.cc:5484] T 3a54e29c093e4be485bf1af1e9ea7a72 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 2 to 3, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:16.938275 9250 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a54e29c093e4be485bf1af1e9ea7a72 with delete_type TABLET_DATA_TOMBSTONED (TS 49adb9e99822494196ed22044bfed81f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55812
I20240501 07:59:16.938486 21497 tablet_replica.cc:330] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: stopping tablet replica
I20240501 07:59:16.938639 21497 raft_consensus.cc:2229] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.938771 21497 raft_consensus.cc:2258] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.939215 21497 ts_tablet_manager.cc:1875] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.941629 21497 ts_tablet_manager.cc:1888] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:16.941697 21497 log.cc:1196] T 3a54e29c093e4be485bf1af1e9ea7a72 P 49adb9e99822494196ed22044bfed81f: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3a54e29c093e4be485bf1af1e9ea7a72
I20240501 07:59:16.943320 6723 catalog_manager.cc:4830] TS 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): tablet 3a54e29c093e4be485bf1af1e9ea7a72 (table rebalancer_extra_table_30 [id=b98f606300d2464cb4085933306f4bf1]) successfully deleted
I20240501 07:59:17.113600 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.126448 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.131311 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:17.133634 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.134627 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:17.143846 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.149438 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.180429 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:17.253134 9665 consensus_queue.cc:237] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:17.254451 6877 raft_consensus.cc:1261] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:17.254482 9534 raft_consensus.cc:1261] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 7fa3a88c25964445975bb47e0feed460: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:17.254884 20648 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:17.255084 20948 consensus_queue.cc:1035] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:17.256940 20648 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:17.257230 9534 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:17.258679 6877 raft_consensus.cc:2932] T b40ef631801d42ec954b19f7323dc00a P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:17.261059 6736 catalog_manager.cc:5484] T b40ef631801d42ec954b19f7323dc00a P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:17.264704 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet b40ef631801d42ec954b19f7323dc00a with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:17.265069 21640 tablet_replica.cc:330] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:17.265314 21640 raft_consensus.cc:2229] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.265519 21640 raft_consensus.cc:2258] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.266515 21640 ts_tablet_manager.cc:1875] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.268909 21640 ts_tablet_manager.cc:1888] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:17.268980 21640 log.cc:1196] T b40ef631801d42ec954b19f7323dc00a P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b40ef631801d42ec954b19f7323dc00a
I20240501 07:59:17.269603 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet b40ef631801d42ec954b19f7323dc00a (table rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]) successfully deleted
I20240501 07:59:17.412293 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:17.413821 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.426438 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:17.432549 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:17.435142 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.442597 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.444126 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.468292 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.522451 9534 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:17.524494 7010 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:17.524746 9271 raft_consensus.cc:1261] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:17.524986 21057 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:17.525189 21689 consensus_queue.cc:1035] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:17.528288 21689 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P ac6c8bc1b74b483791e262abee7862cf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:17.528703 7010 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:17.531857 6735 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:17.535210 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet ee29c58cdc6544e8933300ad5faecb3a with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:17.543236 21713 tablet_replica.cc:330] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:17.543450 21713 raft_consensus.cc:2229] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.543632 21713 raft_consensus.cc:2258] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.544104 21713 ts_tablet_manager.cc:1875] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.544780 9271 raft_consensus.cc:2932] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:17.546567 21713 ts_tablet_manager.cc:1888] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 07:59:17.546653 21713 log.cc:1196] T ee29c58cdc6544e8933300ad5faecb3a P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ee29c58cdc6544e8933300ad5faecb3a
I20240501 07:59:17.549469 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet ee29c58cdc6544e8933300ad5faecb3a (table rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]) successfully deleted
I20240501 07:59:17.692812 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:17.703454 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:17.711836 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.718057 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:17.718417 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.719808 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.727041 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:17.776518 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:17.824313 6878 consensus_queue.cc:237] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:17.825474 9534 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:17.825577 9271 raft_consensus.cc:1261] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:17.825893 21301 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:17.826079 20522 consensus_queue.cc:1035] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:17.828441 21782 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:17.828902 9534 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:17.829151 9271 raft_consensus.cc:2932] T 4882718ff90c4d939917f4970a9acb9e P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:17.832222 6729 catalog_manager.cc:5484] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 reported cstate change: config changed from index 6 to 7, VOTER 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:17.836409 6989 tablet_service.cc:1514] Processing DeleteTablet for tablet 4882718ff90c4d939917f4970a9acb9e with delete_type TABLET_DATA_TOMBSTONED (TS 9a3888360d4d4ace96b6b909c160bfc2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37430
I20240501 07:59:17.836781 21783 tablet_replica.cc:330] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: stopping tablet replica
I20240501 07:59:17.836925 21783 raft_consensus.cc:2229] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.837124 21783 raft_consensus.cc:2258] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.837695 21783 ts_tablet_manager.cc:1875] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.841904 21783 ts_tablet_manager.cc:1888] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 07:59:17.841982 21783 log.cc:1196] T 4882718ff90c4d939917f4970a9acb9e P 9a3888360d4d4ace96b6b909c160bfc2: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4882718ff90c4d939917f4970a9acb9e
I20240501 07:59:17.842793 6724 catalog_manager.cc:4830] TS 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): tablet 4882718ff90c4d939917f4970a9acb9e (table rebalance_test_table_7 [id=74fc7bb11ad34e64bf0ced56aefa12b7]) successfully deleted
I20240501 07:59:17.910944 7271 consensus_queue.cc:237] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:17.912494 7140 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:17.912869 7010 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:17.913211 7207 consensus_peers.cc:485] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e11a47afed3a44afa75f0dd363098c22. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:17.913244 21688 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:17.913702 21688 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:17.915551 21469 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:17.917225 7140 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:17.918218 7010 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:17.919093 6729 catalog_manager.cc:5484] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:17.941677 9403 consensus_queue.cc:237] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:17.943282 9534 raft_consensus.cc:1261] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:17.943404 9337 consensus_peers.cc:485] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 791b17b0049c4bbbbf56934dcfe2f742. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:17.943666 21695 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:17.944415 7271 raft_consensus.cc:1261] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.944928 21695 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:17.948127 21695 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:17.948452 7271 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:17.950451 9534 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:17.951843 6735 catalog_manager.cc:5484] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:17.974764 9403 consensus_queue.cc:237] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:17.976286 9666 raft_consensus.cc:1261] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.976562 6878 raft_consensus.cc:1261] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:17.976799 9338 consensus_peers.cc:485] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 -> Peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797): Couldn't send request to peer 49adb9e99822494196ed22044bfed81f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0cf86e678084bc3b948968e5af0365d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:17.977032 21122 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:17.977279 21695 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:17.979600 21122 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:17.980032 6878 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:17.982657 9666 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:17.983397 6729 catalog_manager.cc:5484] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index -1 to 2, VOTER 49adb9e99822494196ed22044bfed81f (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:18.007550 9534 consensus_queue.cc:237] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
W20240501 07:59:18.009124 9470 consensus_peers.cc:485] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf -> Peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): Couldn't send request to peer fa6964452854415a85ad72f8639768d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a2d5511dc0d4e8cb6d71ca894da4884. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.009187 6878 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.009245 9665 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.009606 21901 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.009809 21057 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.011624 21057 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:18.013346 9666 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:18.013481 6878 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:18.015278 6735 catalog_manager.cc:5484] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:18.039114 7140 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:18.041839 9271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:18.042613 7271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 07:59:18.043058 7076 consensus_peers.cc:485] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be -> Peer 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): Couldn't send request to peer 7fa3a88c25964445975bb47e0feed460. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80152bb09e4e42a09e106cf8b58178e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.043113 21920 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:18.043758 17598 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:18.046872 21920 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:18.047286 7270 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:18.051769 6735 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:18.054925 9271 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:18.077821 9534 consensus_queue.cc:237] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:18.079337 7010 raft_consensus.cc:1261] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.079424 7270 raft_consensus.cc:1261] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.079516 9469 consensus_peers.cc:485] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1d72a9cc6a940b7b15535c8db19c312. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.079710 21901 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.079882 21057 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.082170 21901 raft_consensus.cc:2932] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:18.082593 7270 raft_consensus.cc:2932] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:18.084344 7010 raft_consensus.cc:2932] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:18.085870 6731 catalog_manager.cc:5484] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:18.237920 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.238473 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:18.241772 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:18.254441 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.258608 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.273252 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.285409 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:18.291571 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.411926 22043 ts_tablet_manager.cc:902] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:18.412695 22043 tablet_copy_client.cc:308] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:41309
I20240501 07:59:18.413185 7292 tablet_copy_service.cc:140] P 15ab96f6477b4291932409b80af07213: Received BeginTabletCopySession request for tablet e11a47afed3a44afa75f0dd363098c22 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:49897)
I20240501 07:59:18.413321 7292 tablet_copy_service.cc:161] P 15ab96f6477b4291932409b80af07213: Beginning new tablet copy session on tablet e11a47afed3a44afa75f0dd363098c22 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:49897: session id = fa6964452854415a85ad72f8639768d6-e11a47afed3a44afa75f0dd363098c22
I20240501 07:59:18.414310 7292 tablet_copy_source_session.cc:205] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.416139 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11a47afed3a44afa75f0dd363098c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.429780 22043 tablet_copy_client.cc:783] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.429948 22043 tablet_copy_client.cc:647] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.431596 22043 tablet_copy_client.cc:523] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.432737 22043 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:18.486656 22043 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.487197 22043 tablet_bootstrap.cc:492] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:59:18.487493 22043 ts_tablet_manager.cc:1367] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.055s user 0.022s sys 0.004s
I20240501 07:59:18.488126 22043 raft_consensus.cc:348] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:18.488377 22043 raft_consensus.cc:729] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.488607 22043 consensus_queue.cc:260] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:18.489112 22043 ts_tablet_manager.cc:1398] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.492885 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.493319 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.494326 7292 tablet_copy_service.cc:342] P 15ab96f6477b4291932409b80af07213: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-e11a47afed3a44afa75f0dd363098c22 received from {username='slave'} at 127.0.99.134:49897
I20240501 07:59:18.494400 7292 tablet_copy_service.cc:433] P 15ab96f6477b4291932409b80af07213: ending tablet copy session fa6964452854415a85ad72f8639768d6-e11a47afed3a44afa75f0dd363098c22 on tablet e11a47afed3a44afa75f0dd363098c22 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:59:18.498955 22043 ts_tablet_manager.cc:902] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:18.499501 22043 tablet_copy_client.cc:308] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:18.499982 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:18.501276 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.503106 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet 0a2d5511dc0d4e8cb6d71ca894da4884 from peer fa6964452854415a85ad72f8639768d6 ({username='slave'} at 127.0.99.134:44051)
I20240501 07:59:18.503216 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet 0a2d5511dc0d4e8cb6d71ca894da4884 from peer fa6964452854415a85ad72f8639768d6 at {username='slave'} at 127.0.99.134:44051: session id = fa6964452854415a85ad72f8639768d6-0a2d5511dc0d4e8cb6d71ca894da4884
I20240501 07:59:18.504122 9555 tablet_copy_source_session.cc:205] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.505017 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2d5511dc0d4e8cb6d71ca894da4884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.507377 22043 tablet_copy_client.cc:783] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.507525 22043 tablet_copy_client.cc:647] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.515324 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:18.516054 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:18.522573 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.534917 22065 ts_tablet_manager.cc:902] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Initiating tablet copy from peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:18.535466 22065 tablet_copy_client.cc:308] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37107
I20240501 07:59:18.536005 7160 tablet_copy_service.cc:140] P fc946354b5d04503a56122e59fc320be: Received BeginTabletCopySession request for tablet 80152bb09e4e42a09e106cf8b58178e3 from peer 7fa3a88c25964445975bb47e0feed460 ({username='slave'} at 127.0.99.136:38089)
I20240501 07:59:18.536106 7160 tablet_copy_service.cc:161] P fc946354b5d04503a56122e59fc320be: Beginning new tablet copy session on tablet 80152bb09e4e42a09e106cf8b58178e3 from peer 7fa3a88c25964445975bb47e0feed460 at {username='slave'} at 127.0.99.136:38089: session id = 7fa3a88c25964445975bb47e0feed460-80152bb09e4e42a09e106cf8b58178e3
I20240501 07:59:18.537137 7160 tablet_copy_source_session.cc:205] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.538264 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80152bb09e4e42a09e106cf8b58178e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.540731 22065 tablet_copy_client.cc:783] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.540905 22065 tablet_copy_client.cc:647] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.543109 22043 tablet_copy_client.cc:523] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.544131 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.545464 22043 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Bootstrap starting.
I20240501 07:59:18.551656 22065 tablet_copy_client.cc:523] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.553136 22065 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Bootstrap starting.
I20240501 07:59:18.561105 22072 ts_tablet_manager.cc:902] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:18.561800 22072 tablet_copy_client.cc:308] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:59:18.562345 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet 791b17b0049c4bbbbf56934dcfe2f742 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:56315)
I20240501 07:59:18.562441 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet 791b17b0049c4bbbbf56934dcfe2f742 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:56315: session id = fc946354b5d04503a56122e59fc320be-791b17b0049c4bbbbf56934dcfe2f742
I20240501 07:59:18.563424 9414 tablet_copy_source_session.cc:205] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.564533 22072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791b17b0049c4bbbbf56934dcfe2f742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.576814 22043 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.577117 22043 tablet_bootstrap.cc:492] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Bootstrap complete.
I20240501 07:59:18.577268 22043 ts_tablet_manager.cc:1367] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240501 07:59:18.577723 22043 raft_consensus.cc:348] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:18.577834 22043 raft_consensus.cc:729] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.577944 22043 consensus_queue.cc:260] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:18.578198 22043 ts_tablet_manager.cc:1398] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.579094 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fa6964452854415a85ad72f8639768d6-0a2d5511dc0d4e8cb6d71ca894da4884 received from {username='slave'} at 127.0.99.134:44051
I20240501 07:59:18.579180 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fa6964452854415a85ad72f8639768d6-0a2d5511dc0d4e8cb6d71ca894da4884 on tablet 0a2d5511dc0d4e8cb6d71ca894da4884 with peer fa6964452854415a85ad72f8639768d6
I20240501 07:59:18.580511 22072 tablet_copy_client.cc:783] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.580700 22072 tablet_copy_client.cc:647] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.584483 22077 ts_tablet_manager.cc:902] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Initiating tablet copy from peer fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:18.584998 22077 tablet_copy_client.cc:308] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:42153
I20240501 07:59:18.585371 9414 tablet_copy_service.cc:140] P fa6964452854415a85ad72f8639768d6: Received BeginTabletCopySession request for tablet c0cf86e678084bc3b948968e5af0365d from peer 49adb9e99822494196ed22044bfed81f ({username='slave'} at 127.0.99.133:40645)
I20240501 07:59:18.585453 9414 tablet_copy_service.cc:161] P fa6964452854415a85ad72f8639768d6: Beginning new tablet copy session on tablet c0cf86e678084bc3b948968e5af0365d from peer 49adb9e99822494196ed22044bfed81f at {username='slave'} at 127.0.99.133:40645: session id = 49adb9e99822494196ed22044bfed81f-c0cf86e678084bc3b948968e5af0365d
I20240501 07:59:18.586346 9414 tablet_copy_source_session.cc:205] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.588579 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0cf86e678084bc3b948968e5af0365d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.592259 22072 tablet_copy_client.cc:523] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.593804 22077 tablet_copy_client.cc:783] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.593967 22077 tablet_copy_client.cc:647] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.594331 22072 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:18.614691 22065 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.615056 22065 tablet_bootstrap.cc:492] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Bootstrap complete.
I20240501 07:59:18.615228 22065 ts_tablet_manager.cc:1367] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.004s
I20240501 07:59:18.615278 22077 tablet_copy_client.cc:523] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.615667 22065 raft_consensus.cc:348] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:18.615819 22065 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.615962 22065 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:18.616261 22065 ts_tablet_manager.cc:1398] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.616973 22077 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Bootstrap starting.
I20240501 07:59:18.618686 7160 tablet_copy_service.cc:342] P fc946354b5d04503a56122e59fc320be: Request end of tablet copy session 7fa3a88c25964445975bb47e0feed460-80152bb09e4e42a09e106cf8b58178e3 received from {username='slave'} at 127.0.99.136:38089
I20240501 07:59:18.618764 7160 tablet_copy_service.cc:433] P fc946354b5d04503a56122e59fc320be: ending tablet copy session 7fa3a88c25964445975bb47e0feed460-80152bb09e4e42a09e106cf8b58178e3 on tablet 80152bb09e4e42a09e106cf8b58178e3 with peer 7fa3a88c25964445975bb47e0feed460
I20240501 07:59:18.621912 22072 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.621906 22077 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.622159 22077 tablet_bootstrap.cc:492] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Bootstrap complete.
I20240501 07:59:18.622159 22072 tablet_bootstrap.cc:492] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:18.622301 22072 ts_tablet_manager.cc:1367] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240501 07:59:18.622301 22077 ts_tablet_manager.cc:1367] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:18.622941 22077 raft_consensus.cc:348] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:18.623088 22077 raft_consensus.cc:729] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.623234 22077 consensus_queue.cc:260] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:18.623544 22077 ts_tablet_manager.cc:1398] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.625711 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session 49adb9e99822494196ed22044bfed81f-c0cf86e678084bc3b948968e5af0365d received from {username='slave'} at 127.0.99.133:40645
I20240501 07:59:18.625799 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session 49adb9e99822494196ed22044bfed81f-c0cf86e678084bc3b948968e5af0365d on tablet c0cf86e678084bc3b948968e5af0365d with peer 49adb9e99822494196ed22044bfed81f
I20240501 07:59:18.626839 22072 raft_consensus.cc:348] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:18.626981 22072 raft_consensus.cc:729] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.628005 22072 consensus_queue.cc:260] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:18.628441 22072 ts_tablet_manager.cc:1398] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:18.629472 9414 tablet_copy_service.cc:342] P fa6964452854415a85ad72f8639768d6: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-791b17b0049c4bbbbf56934dcfe2f742 received from {username='slave'} at 127.0.99.131:56315
I20240501 07:59:18.629562 9414 tablet_copy_service.cc:433] P fa6964452854415a85ad72f8639768d6: ending tablet copy session fc946354b5d04503a56122e59fc320be-791b17b0049c4bbbbf56934dcfe2f742 on tablet 791b17b0049c4bbbbf56934dcfe2f742 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:18.677754 22072 ts_tablet_manager.cc:902] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:18.678194 22072 tablet_copy_client.cc:308] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
I20240501 07:59:18.679208 9555 tablet_copy_service.cc:140] P ac6c8bc1b74b483791e262abee7862cf: Received BeginTabletCopySession request for tablet a1d72a9cc6a940b7b15535c8db19c312 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:43037)
I20240501 07:59:18.679318 9555 tablet_copy_service.cc:161] P ac6c8bc1b74b483791e262abee7862cf: Beginning new tablet copy session on tablet a1d72a9cc6a940b7b15535c8db19c312 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:43037: session id = fc946354b5d04503a56122e59fc320be-a1d72a9cc6a940b7b15535c8db19c312
I20240501 07:59:18.680292 9555 tablet_copy_source_session.cc:205] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.682389 22072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d72a9cc6a940b7b15535c8db19c312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.685021 22072 tablet_copy_client.cc:783] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.685225 22072 tablet_copy_client.cc:647] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.688865 22072 tablet_copy_client.cc:523] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.690110 22072 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:18.695566 22072 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.695854 22072 tablet_bootstrap.cc:492] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:18.696010 22072 ts_tablet_manager.cc:1367] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 07:59:18.696476 22072 raft_consensus.cc:348] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:18.696578 22072 raft_consensus.cc:729] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.696833 22072 consensus_queue.cc:260] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:18.697520 22072 ts_tablet_manager.cc:1398] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:18.698385 9555 tablet_copy_service.cc:342] P ac6c8bc1b74b483791e262abee7862cf: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-a1d72a9cc6a940b7b15535c8db19c312 received from {username='slave'} at 127.0.99.131:43037
I20240501 07:59:18.698446 9555 tablet_copy_service.cc:433] P ac6c8bc1b74b483791e262abee7862cf: ending tablet copy session fc946354b5d04503a56122e59fc320be-a1d72a9cc6a940b7b15535c8db19c312 on tablet a1d72a9cc6a940b7b15535c8db19c312 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:18.774346 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.781524 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.785794 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:18.803470 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.803980 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:18.804791 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:18.807962 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:18.815065 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:18.837589 9403 raft_consensus.cc:1203] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:18.955374 9266 raft_consensus.cc:1203] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.000108 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.000305 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:19.006549 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.016260 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:19.016778 9403 raft_consensus.cc:1203] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.020852 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.021606 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.026901 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.039436 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:19.092883 7140 raft_consensus.cc:1203] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.137712 9664 raft_consensus.cc:1203] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 07:59:19.236698 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.243631 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:19.247516 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.259747 7140 raft_consensus.cc:1203] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.267203 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.269227 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:19.270004 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.275965 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:19.310596 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.354698 9534 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884"
dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40846
I20240501 07:59:19.354851 9534 raft_consensus.cc:595] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:19.509514 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.514703 9515 tablet_service.cc:1429] Tablet server has 26 leaders and 0 scanners
I20240501 07:59:19.522972 21689 raft_consensus.cc:979] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: : Instructing follower fa6964452854415a85ad72f8639768d6 to start an election
I20240501 07:59:19.523077 21689 raft_consensus.cc:1067] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Signalling peer fa6964452854415a85ad72f8639768d6 to start an election
I20240501 07:59:19.523475 9401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884"
dest_uuid: "fa6964452854415a85ad72f8639768d6"
from {username='slave'} at 127.0.99.135:51085
I20240501 07:59:19.523633 9401 raft_consensus.cc:482] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:19.523695 9401 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:19.524868 9401 raft_consensus.cc:504] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:19.524927 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:19.525305 9401 leader_election.cc:288] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 election: Requested vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:19.525566 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:19.525601 6877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:19.525679 9664 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:19.525739 6877 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:19.526727 9664 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 2.
I20240501 07:59:19.526820 6877 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 2.
I20240501 07:59:19.526942 9534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2d5511dc0d4e8cb6d71ca894da4884" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:19.527102 9534 raft_consensus.cc:3032] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:19.527135 9534 raft_consensus.cc:729] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:19.527278 9534 consensus_queue.cc:260] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:19.527441 9534 raft_consensus.cc:3037] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:19.528360 9534 raft_consensus.cc:2454] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 2.
I20240501 07:59:19.528678 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:19.528828 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.531183 9338 leader_election.cc:302] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:19.531354 21695 raft_consensus.cc:2781] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:19.531383 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.531416 21695 raft_consensus.cc:686] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 2 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:19.531538 21695 consensus_queue.cc:237] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:19.533411 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:19.535351 6735 catalog_manager.cc:5484] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:19.561061 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.811707 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.811736 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:19.833701 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:19.834380 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.834411 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:19.851585 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:19.852579 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:19.861841 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:19.930847 9665 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:19.931265 21884 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:19.937784 9401 consensus_queue.cc:237] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:19.938946 7271 raft_consensus.cc:1261] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:19.939100 7140 raft_consensus.cc:1261] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:19.939469 21695 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:19.939638 21884 consensus_queue.cc:1035] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:19.941478 22373 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:19.941833 7140 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:19.941923 7271 raft_consensus.cc:2932] T 791b17b0049c4bbbbf56934dcfe2f742 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:19.945223 6731 catalog_manager.cc:5484] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:19.948418 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 791b17b0049c4bbbbf56934dcfe2f742 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:19.948946 22374 tablet_replica.cc:330] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:19.949127 22374 raft_consensus.cc:2229] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:19.949323 22374 raft_consensus.cc:2258] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:19.949833 22374 ts_tablet_manager.cc:1875] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:19.953557 22374 ts_tablet_manager.cc:1888] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:19.953677 22374 log.cc:1196] T 791b17b0049c4bbbbf56934dcfe2f742 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/791b17b0049c4bbbbf56934dcfe2f742
I20240501 07:59:19.954818 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 791b17b0049c4bbbbf56934dcfe2f742 (table rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]) successfully deleted
I20240501 07:59:20.024888 6878 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:20.025315 21695 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.032604 9534 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:20.033488 21884 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.100862 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.106117 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.109017 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:20.117053 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:20.117138 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:20.117379 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:20.122181 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.123571 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:20.210773 7271 consensus_queue.cc:237] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:20.212716 9403 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:20.212728 7010 raft_consensus.cc:1261] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:20.213148 22342 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:20.213564 22342 consensus_queue.cc:1035] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:20.215989 22231 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:20.216351 9403 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:20.222429 6732 catalog_manager.cc:5484] T e11a47afed3a44afa75f0dd363098c22 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:20.225888 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet e11a47afed3a44afa75f0dd363098c22 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:20.231487 22455 tablet_replica.cc:330] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:20.231709 22455 raft_consensus.cc:2229] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:20.231885 22455 raft_consensus.cc:2258] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:20.232266 7009 raft_consensus.cc:2932] T e11a47afed3a44afa75f0dd363098c22 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:20.232663 22455 ts_tablet_manager.cc:1875] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:20.235186 22455 ts_tablet_manager.cc:1888] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:20.241822 22455 log.cc:1196] T e11a47afed3a44afa75f0dd363098c22 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e11a47afed3a44afa75f0dd363098c22
I20240501 07:59:20.242376 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet e11a47afed3a44afa75f0dd363098c22 (table rebalance_test_table_19 [id=ea54f4fba04447f2a7ba1c5bf0a4fb37]) successfully deleted
I20240501 07:59:20.364614 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:20.374115 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.375617 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.378897 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:20.384354 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.389612 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:20.392299 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:20.398263 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:20.481361 9403 consensus_queue.cc:237] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:20.482940 9271 raft_consensus.cc:1261] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.483225 6876 raft_consensus.cc:1261] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.483655 21695 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.483855 21884 consensus_queue.cc:1035] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.485774 22418 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:20.486222 9271 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:20.486295 6876 raft_consensus.cc:2932] T c0cf86e678084bc3b948968e5af0365d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:20.489161 6735 catalog_manager.cc:5484] T c0cf86e678084bc3b948968e5af0365d P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:20.492486 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet c0cf86e678084bc3b948968e5af0365d with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:20.492828 22526 tablet_replica.cc:330] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:20.493005 22526 raft_consensus.cc:2229] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:20.493276 22526 raft_consensus.cc:2258] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:20.493745 22526 ts_tablet_manager.cc:1875] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:20.497493 22526 ts_tablet_manager.cc:1888] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:20.497786 22526 log.cc:1196] T c0cf86e678084bc3b948968e5af0365d P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/c0cf86e678084bc3b948968e5af0365d
I20240501 07:59:20.498376 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet c0cf86e678084bc3b948968e5af0365d (table rebalancer_extra_table_24 [id=bdf877948fca4d2693052c9c2f39a865]) successfully deleted
I20240501 07:59:20.640838 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.641537 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:20.647992 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:20.650831 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.653892 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.675477 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:20.677625 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:20.696866 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:20.742846 9534 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:20.744201 7010 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:20.744203 7140 raft_consensus.cc:1261] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:20.744613 21689 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:20.744828 21901 consensus_queue.cc:1035] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:20.746696 22596 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:20.747104 7139 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:20.747462 7010 raft_consensus.cc:2932] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:20.750635 6731 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P ac6c8bc1b74b483791e262abee7862cf reported cstate change: config changed from index 4 to 5, VOTER 7fa3a88c25964445975bb47e0feed460 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:20.754895 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet f8921a8351e04daba09a8e7e4f683fd9 with delete_type TABLET_DATA_TOMBSTONED (TS 7fa3a88c25964445975bb47e0feed460 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33078
I20240501 07:59:20.755175 22526 tablet_replica.cc:330] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: stopping tablet replica
I20240501 07:59:20.755336 22526 raft_consensus.cc:2229] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:20.755477 22526 raft_consensus.cc:2258] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:20.755932 22526 ts_tablet_manager.cc:1875] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:20.758141 22526 ts_tablet_manager.cc:1888] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:20.758226 22526 log.cc:1196] T f8921a8351e04daba09a8e7e4f683fd9 P 7fa3a88c25964445975bb47e0feed460: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f8921a8351e04daba09a8e7e4f683fd9
I20240501 07:59:20.758720 6723 catalog_manager.cc:4830] TS 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111): tablet f8921a8351e04daba09a8e7e4f683fd9 (table rebalancer_extra_table_8 [id=d88c9b6fb58b4b6aab44b2d11474f3e3]) successfully deleted
I20240501 07:59:20.914435 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.915171 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:20.916224 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.916239 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:20.917153 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:20.919195 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:20.932268 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:20.941135 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.006459 9403 consensus_queue.cc:237] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:21.007956 9665 raft_consensus.cc:1261] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:21.007956 7010 raft_consensus.cc:1261] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:21.008370 21884 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.008623 22418 consensus_queue.cc:1035] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.010658 21884 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:21.011502 9665 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:21.012391 7009 raft_consensus.cc:2932] T 2b09efd2448a4be7b63636b66e56a030 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:21.015193 6730 catalog_manager.cc:5484] T 2b09efd2448a4be7b63636b66e56a030 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 3, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:21.018347 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b09efd2448a4be7b63636b66e56a030 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:21.018958 22671 tablet_replica.cc:330] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:21.019172 22671 raft_consensus.cc:2229] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:21.019311 22671 raft_consensus.cc:2258] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:21.019757 22671 ts_tablet_manager.cc:1875] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:21.022706 22671 ts_tablet_manager.cc:1888] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:21.022786 22671 log.cc:1196] T 2b09efd2448a4be7b63636b66e56a030 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2b09efd2448a4be7b63636b66e56a030
I20240501 07:59:21.023411 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 2b09efd2448a4be7b63636b66e56a030 (table rebalancer_extra_table_16 [id=ed5bc81d8f96454b8c9f400c29fb9ea6]) successfully deleted
I20240501 07:59:21.156576 6857 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:21.158077 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.168314 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:21.172067 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.188629 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.204020 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.210927 9382 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:21.226945 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:21.265400 9403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b78be08b353141469d49c7460393a051"
dest_uuid: "fa6964452854415a85ad72f8639768d6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49006
I20240501 07:59:21.265576 9403 raft_consensus.cc:595] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:21.276036 22418 raft_consensus.cc:979] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: : Instructing follower 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:59:21.276157 22418 raft_consensus.cc:1067] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Signalling peer 1a9029bee6a144938f68b363e4cca162 to start an election
I20240501 07:59:21.276736 6876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b78be08b353141469d49c7460393a051"
dest_uuid: "1a9029bee6a144938f68b363e4cca162"
from {username='slave'} at 127.0.99.134:47957
I20240501 07:59:21.276859 6876 raft_consensus.cc:482] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:21.276921 6876 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.278041 6876 raft_consensus.cc:504] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:21.278558 6876 leader_election.cc:288] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:21.278868 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:21.278986 7009 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.279383 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:21.279419 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78be08b353141469d49c7460393a051" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:21.279500 9403 raft_consensus.cc:3032] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:21.279541 9403 raft_consensus.cc:729] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:21.279538 9266 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.279706 9403 consensus_queue.cc:260] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:21.279937 9403 raft_consensus.cc:3037] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.280102 7009 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:21.280483 9266 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:21.280771 6814 leader_election.cc:302] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:21.280938 21301 raft_consensus.cc:2781] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:21.280962 9403 raft_consensus.cc:2454] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:21.281039 21301 raft_consensus.cc:686] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:21.281205 21301 consensus_queue.cc:237] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:21.283746 6728 catalog_manager.cc:5484] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from fa6964452854415a85ad72f8639768d6 (127.0.99.134) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:21.427868 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.430748 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:21.432870 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:21.434418 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.436764 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.436990 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.445400 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.464339 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:21.591274 9266 consensus_queue.cc:237] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:21.593135 9665 raft_consensus.cc:1261] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:21.593155 9403 raft_consensus.cc:1261] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:21.593473 22820 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:21.593679 9205 consensus_peers.cc:485] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72937d60b2014cb490cbbb89c1082519. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.593684 22243 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:21.595346 22243 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:21.595795 9665 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:21.598493 9403 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:21.599491 6737 catalog_manager.cc:5484] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:21.732515 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.733932 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:21.735363 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.749169 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:21.757236 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:21.768579 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.773630 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.783120 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.820789 9403 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.821255 22579 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.847295 9266 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.847652 22579 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.852715 7009 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.853093 21301 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.968488 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:21.969388 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.969677 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.970927 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:21.975723 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:21.996239 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:22.009733 9515 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240501 07:59:22.024586 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:22.059079 9535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312"
dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41000
I20240501 07:59:22.059228 9535 raft_consensus.cc:595] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:22.122273 21689 raft_consensus.cc:979] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf: : Instructing follower fc946354b5d04503a56122e59fc320be to start an election
I20240501 07:59:22.122437 21689 raft_consensus.cc:1067] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Signalling peer fc946354b5d04503a56122e59fc320be to start an election
I20240501 07:59:22.122954 7140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312"
dest_uuid: "fc946354b5d04503a56122e59fc320be"
from {username='slave'} at 127.0.99.135:53675
I20240501 07:59:22.123155 7140 raft_consensus.cc:482] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:22.123222 7140 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.124341 7140 raft_consensus.cc:504] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:22.124917 7140 leader_election.cc:288] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:22.126654 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:22.126786 7009 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.128011 7009 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:22.126945 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ac6c8bc1b74b483791e262abee7862cf"
I20240501 07:59:22.128634 9535 raft_consensus.cc:3032] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:22.128734 9535 raft_consensus.cc:729] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac6c8bc1b74b483791e262abee7862cf, State: Running, Role: LEADER
I20240501 07:59:22.128805 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d72a9cc6a940b7b15535c8db19c312" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:22.128921 7271 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.128888 9535 consensus_queue.cc:260] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:22.129117 9535 raft_consensus.cc:3037] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.129969 7271 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:22.130204 9535 raft_consensus.cc:2454] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:22.130362 7075 leader_election.cc:302] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:22.130548 22777 raft_consensus.cc:2781] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:22.130628 22777 raft_consensus.cc:686] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [term 2 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:22.130761 22777 consensus_queue.cc:237] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:22.137639 6728 catalog_manager.cc:5484] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:22.175210 22984 ts_tablet_manager.cc:902] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:22.175760 22984 tablet_copy_client.cc:308] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46797
I20240501 07:59:22.176259 9291 tablet_copy_service.cc:140] P 49adb9e99822494196ed22044bfed81f: Received BeginTabletCopySession request for tablet 72937d60b2014cb490cbbb89c1082519 from peer fc946354b5d04503a56122e59fc320be ({username='slave'} at 127.0.99.131:44503)
I20240501 07:59:22.176375 9291 tablet_copy_service.cc:161] P 49adb9e99822494196ed22044bfed81f: Beginning new tablet copy session on tablet 72937d60b2014cb490cbbb89c1082519 from peer fc946354b5d04503a56122e59fc320be at {username='slave'} at 127.0.99.131:44503: session id = fc946354b5d04503a56122e59fc320be-72937d60b2014cb490cbbb89c1082519
I20240501 07:59:22.177441 9291 tablet_copy_source_session.cc:205] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:22.178318 22984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72937d60b2014cb490cbbb89c1082519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:22.188062 22984 tablet_copy_client.cc:783] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:22.188266 22984 tablet_copy_client.cc:647] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:22.189901 22984 tablet_copy_client.cc:523] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:22.191303 22984 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Bootstrap starting.
I20240501 07:59:22.197870 22984 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:22.198151 22984 tablet_bootstrap.cc:492] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Bootstrap complete.
I20240501 07:59:22.198352 22984 ts_tablet_manager.cc:1367] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 07:59:22.198820 22984 raft_consensus.cc:348] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:22.198963 22984 raft_consensus.cc:729] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Initialized, Role: FOLLOWER
I20240501 07:59:22.200043 22984 consensus_queue.cc:260] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:22.200462 22984 ts_tablet_manager.cc:1398] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:22.201107 9291 tablet_copy_service.cc:342] P 49adb9e99822494196ed22044bfed81f: Request end of tablet copy session fc946354b5d04503a56122e59fc320be-72937d60b2014cb490cbbb89c1082519 received from {username='slave'} at 127.0.99.131:44503
I20240501 07:59:22.201184 9291 tablet_copy_service.cc:433] P 49adb9e99822494196ed22044bfed81f: ending tablet copy session fc946354b5d04503a56122e59fc320be-72937d60b2014cb490cbbb89c1082519 on tablet 72937d60b2014cb490cbbb89c1082519 with peer fc946354b5d04503a56122e59fc320be
I20240501 07:59:22.211858 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.220430 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:22.223091 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.224716 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:22.225611 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.236852 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:22.251930 7121 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:22.278122 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:22.474495 7250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.475190 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.475579 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:22.475649 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:22.484789 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.485766 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:22.491833 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:22.502328 7121 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:22.561283 7271 raft_consensus.cc:1261] T a1d72a9cc6a940b7b15535c8db19c312 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:22.561717 22189 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:22.578876 7140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3"
dest_uuid: "fc946354b5d04503a56122e59fc320be"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43692
I20240501 07:59:22.579046 7140 raft_consensus.cc:595] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:22.599684 22777 raft_consensus.cc:979] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: : Instructing follower 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:59:22.599797 22777 raft_consensus.cc:1067] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Signalling peer 15ab96f6477b4291932409b80af07213 to start an election
I20240501 07:59:22.600203 7271 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3"
dest_uuid: "15ab96f6477b4291932409b80af07213"
from {username='slave'} at 127.0.99.131:49253
I20240501 07:59:22.600368 7271 raft_consensus.cc:482] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:22.600430 7271 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.601552 7271 raft_consensus.cc:504] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:22.602102 7271 leader_election.cc:288] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:22.602568 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:22.602572 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:22.602689 9266 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.602697 7140 raft_consensus.cc:3032] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:22.602735 7140 raft_consensus.cc:729] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:22.602885 7140 consensus_queue.cc:260] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:22.603113 7140 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.603734 9266 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:22.603899 7140 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:22.604192 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152bb09e4e42a09e106cf8b58178e3" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:22.604229 7206 leader_election.cc:302] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:22.604332 9664 raft_consensus.cc:3037] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:22.604451 22231 raft_consensus.cc:2781] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:22.604554 22231 raft_consensus.cc:686] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:22.604708 22231 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:22.605425 9664 raft_consensus.cc:2454] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:22.608168 6736 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from fc946354b5d04503a56122e59fc320be (127.0.99.131) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:22.672133 7009 raft_consensus.cc:1261] T a1d72a9cc6a940b7b15535c8db19c312 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:22.672577 22777 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:22.676729 9534 raft_consensus.cc:1261] T a1d72a9cc6a940b7b15535c8db19c312 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Refusing update from remote peer fc946354b5d04503a56122e59fc320be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:22.677386 22777 consensus_queue.cc:1035] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:22.707479 7140 raft_consensus.cc:1203] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:22.721482 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:22.726209 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:22.729579 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:22.735260 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.741487 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:22.751441 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:22.758206 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.758456 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:22.953701 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:22.956354 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:22.964512 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.966670 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:22.979578 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:22.991724 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:22.992367 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.012125 7140 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:23.012717 22231 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:23.021807 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.068589 9403 consensus_queue.cc:237] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:23.069891 6876 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:23.069904 9664 raft_consensus.cc:1261] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer fa6964452854415a85ad72f8639768d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:23.070246 21884 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.070463 22418 consensus_queue.cc:1035] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.072417 23186 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:23.072558 6876 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:23.072645 9664 raft_consensus.cc:2932] T 0a2d5511dc0d4e8cb6d71ca894da4884 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:23.076489 6734 catalog_manager.cc:5484] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 reported cstate change: config changed from index 2 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) evicted. New cstate: current_term: 2 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:23.079418 9515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2d5511dc0d4e8cb6d71ca894da4884 with delete_type TABLET_DATA_TOMBSTONED (TS ac6c8bc1b74b483791e262abee7862cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46788
I20240501 07:59:23.080219 23189 tablet_replica.cc:330] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: stopping tablet replica
I20240501 07:59:23.080581 23189 raft_consensus.cc:2229] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:23.080780 23189 raft_consensus.cc:2258] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:23.081395 23189 ts_tablet_manager.cc:1875] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:23.084643 23189 ts_tablet_manager.cc:1888] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 07:59:23.084779 23189 log.cc:1196] T 0a2d5511dc0d4e8cb6d71ca894da4884 P ac6c8bc1b74b483791e262abee7862cf: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0a2d5511dc0d4e8cb6d71ca894da4884
I20240501 07:59:23.085328 6724 catalog_manager.cc:4830] TS ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): tablet 0a2d5511dc0d4e8cb6d71ca894da4884 (table rebalancer_extra_table_2 [id=7b930b491bfc45b296e95dda90955ff6]) successfully deleted
I20240501 07:59:23.165367 9271 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:23.166394 22231 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:23.172785 9665 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:23.175726 22876 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:23.260530 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:23.275907 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:23.277252 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.284195 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.284356 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:23.284452 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.287194 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:23.293036 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.371706 6878 consensus_queue.cc:237] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:23.372946 7009 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:23.372964 9266 raft_consensus.cc:1261] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 1a9029bee6a144938f68b363e4cca162: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:23.373461 22579 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.373647 22891 consensus_queue.cc:1035] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.376060 23296 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:23.376559 7009 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:23.376732 9266 raft_consensus.cc:2932] T b78be08b353141469d49c7460393a051 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:23.380633 6735 catalog_manager.cc:5484] T b78be08b353141469d49c7460393a051 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index 2 to 4, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:23.383152 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet b78be08b353141469d49c7460393a051 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:23.385164 23297 tablet_replica.cc:330] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:23.385308 23297 raft_consensus.cc:2229] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:23.385416 23297 raft_consensus.cc:2258] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:23.385816 23297 ts_tablet_manager.cc:1875] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:23.388087 23297 ts_tablet_manager.cc:1888] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 07:59:23.388201 23297 log.cc:1196] T b78be08b353141469d49c7460393a051 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b78be08b353141469d49c7460393a051
I20240501 07:59:23.388742 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet b78be08b353141469d49c7460393a051 (table rebalancer_extra_table_36 [id=1ab5ad387dbd4009a30d29d365a32c87]) successfully deleted
I20240501 07:59:23.417371 9534 consensus_queue.cc:237] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:23.418937 9403 raft_consensus.cc:1261] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:23.419044 9664 raft_consensus.cc:1261] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer ac6c8bc1b74b483791e262abee7862cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:23.419395 9469 consensus_peers.cc:485] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d818b9a4d9d48d9bab2b1c621db6828. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:23.419550 23314 consensus_queue.cc:1035] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:23.419742 22596 consensus_queue.cc:1035] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:23.421603 23314 raft_consensus.cc:2932] T 0d818b9a4d9d48d9bab2b1c621db6828 P ac6c8bc1b74b483791e262abee7862cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.421981 9664 raft_consensus.cc:2932] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.425877 9403 raft_consensus.cc:2932] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.425942 6736 catalog_manager.cc:5484] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "ac6c8bc1b74b483791e262abee7862cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.448745 7271 consensus_queue.cc:237] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:23.450146 6878 raft_consensus.cc:1261] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:23.450193 7139 raft_consensus.cc:1261] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:23.450461 7208 consensus_peers.cc:485] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9bd69f172f440a98debac09a97b1f8d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:23.450647 23255 consensus_queue.cc:1035] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.450871 22876 consensus_queue.cc:1035] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:23.452862 22876 raft_consensus.cc:2932] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:23.453255 6878 raft_consensus.cc:2932] T a9bd69f172f440a98debac09a97b1f8d P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:23.455204 7139 raft_consensus.cc:2932] T a9bd69f172f440a98debac09a97b1f8d P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:23.456645 6736 catalog_manager.cc:5484] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 reported cstate change: config changed from index -1 to 4, VOTER ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:23.615727 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:23.617225 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.617734 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:23.628942 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:23.633962 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.634207 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.645308 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:23.654618 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.848443 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:23.851889 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.856580 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:23.860478 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:23.868780 9515 tablet_service.cc:1429] Tablet server has 24 leaders and 0 scanners
I20240501 07:59:23.885949 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.890553 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:23.922152 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:23.925470 23440 ts_tablet_manager.cc:902] T a9bd69f172f440a98debac09a97b1f8d P ac6c8bc1b74b483791e262abee7862cf: Initiating tablet copy from peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
F20240501 07:59:23.925678 23440 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.queue_time_us, tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.queue_time_us) (tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.run_cpu_time_us, tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.run_cpu_time_us) (tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.run_wall_time_us, tablet-download-pool-2ba12580263f4bd883ad7e003ebd6c51.run_wall_time_us) (tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.queue_time_us, tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.queue_time_us) (tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.run_cpu_time_us, tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.run_cpu_time_us) (tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.run_wall_time_us, tablet-download-pool-33e711b7a9684137969ce2bbffae40c2.run_wall_time_us) (tablet-download-pool-3839146ca1f0452499eb6b5083097d00.queue_time_us, tablet-download-pool-3839146ca1f0452499eb6b5083097d00.queue_time_us) (tablet-download-pool-3839146ca1f0452499eb6b5083097d00.run_cpu_time_us, tablet-download-pool-3839146ca1f0452499eb6b5083097d00.run_cpu_time_us) (tablet-download-pool-3839146ca1f0452499eb6b5083097d00.run_wall_time_us, tablet-download-pool-3839146ca1f0452499eb6b5083097d00.run_wall_time_us) (tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.queue_time_us, tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.queue_time_us) (tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.run_cpu_time_us, tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.run_cpu_time_us) (tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.run_wall_time_us, tablet-download-pool-4882718ff90c4d939917f4970a9acb9e.run_wall_time_us) (tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.queue_time_us, tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.queue_time_us) (tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.run_cpu_time_us, tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.run_cpu_time_us) (tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.run_wall_time_us, tablet-download-pool-48964a6c57584c3eacc6dffac29d77a8.run_wall_time_us) (tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.queue_time_us, tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.queue_time_us) (tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.run_cpu_time_us, tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.run_cpu_time_us) (tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.run_wall_time_us, tablet-download-pool-4f3e4cd27e4545baa255a01023cf1d68.run_wall_time_us) (tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.queue_time_us, tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.queue_time_us) (tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.run_cpu_time_us, tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.run_cpu_time_us) (tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.run_wall_time_us, tablet-download-pool-69ff85c89fa443539b28c0d0cc299cfd.run_wall_time_us) (tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.queue_time_us, tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.queue_time_us) (tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.run_cpu_time_us, tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.run_cpu_time_us) (tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.run_wall_time_us, tablet-download-pool-8c0f4ea5ccba462d9468e3f445569eaa.run_wall_time_us) (tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.queue_time_us, tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.queue_time_us) (tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.run_cpu_time_us, tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.run_cpu_time_us) (tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.run_wall_time_us, tablet-download-pool-9051ba90bad3430fa2e128c88ec7b6f4.run_wall_time_us) (tablet-download-pool-a9bd69f172f440a98debac09a97b1f8d.queue_time_us, tablet-download-pool-a9bd69f172f440a98debac09a97b1f8d.queue_time_us) (tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.queue_time_us, tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.queue_time_us) (tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.run_cpu_time_us, tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.run_cpu_time_us) (tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.run_wall_time_us, tablet-download-pool-ab7dcdd38eff43fe9fff9081ee644c8d.run_wall_time_us) (tablet-download-pool-c80723cb0ce4446483a5f1754e066081.queue_time_us, tablet-download-pool-c80723cb0ce4446483a5f1754e066081.queue_time_us) (tablet-download-pool-c80723cb0ce4446483a5f1754e066081.run_cpu_time_us, tablet-download-pool-c80723cb0ce4446483a5f1754e066081.run_cpu_time_us) (tablet-download-pool-c80723cb0ce4446483a5f1754e066081.run_wall_time_us, tablet-download-pool-c80723cb0ce4446483a5f1754e066081.run_wall_time_us) (tablet-download-pool-e05498632f0945c29f723a4613a26a04.queue_time_us, tablet-download-pool-e05498632f0945c29f723a4613a26a04.queue_time_us) (tablet-download-pool-e05498632f0945c29f723a4613a26a04.run_cpu_time_us, tablet-download-pool-e05498632f0945c29f723a4613a26a04.run_cpu_time_us) (tablet-download-pool-e05498632f0945c29f723a4613a26a04.run_wall_time_us, tablet-download-pool-e05498632f0945c29f723a4613a26a04.run_wall_time_us) (tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.queue_time_us, tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.queue_time_us) (tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.run_cpu_time_us, tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.run_cpu_time_us) (tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.run_wall_time_us, tablet-download-pool-f334b66a84b94c3ba15004253b4cb714.run_wall_time_us) (tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.queue_time_us, tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.queue_time_us) (tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.run_cpu_time_us, tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.run_cpu_time_us) (tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.run_wall_time_us, tablet-download-pool-f723d182d04d439da6ab5b811448cf2d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f33fefc9dcd google::LogMessage::Fail() at ??:0
@ 0x7f33fefcdb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f33fefc97cc google::LogMessage::Flush() at ??:0
@ 0x7f33fefcaf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3400433044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3400417f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f3400415efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f3402fc877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3402fd1a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f340305276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f34030664f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3403051226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f3403060c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5579c875e75c std::function<>::operator()() at ??:0
@ 0x7f340041ae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f340041b6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f340041d03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5579c875e75c std::function<>::operator()() at ??:0
I20240501 07:59:23.986707 9266 consensus_queue.cc:237] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:23.988077 7140 raft_consensus.cc:1261] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7f340040a807 kudu::Thread::SuperviseThread() at ??:0
I20240501 07:59:23.988507 9665 raft_consensus.cc:1261] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 49adb9e99822494196ed22044bfed81f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:23.988484 23155 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:23.989423 22820 consensus_queue.cc:1035] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7f3400c486db start_thread at ??:0
I20240501 07:59:23.991582 23467 ts_tablet_manager.cc:902] T 0d818b9a4d9d48d9bab2b1c621db6828 P fc946354b5d04503a56122e59fc320be: Initiating tablet copy from peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:23.991456 23155 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.992066 23467 tablet_copy_client.cc:308] T 0d818b9a4d9d48d9bab2b1c621db6828 P fc946354b5d04503a56122e59fc320be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:36013
@ 0x7f33fe18071f clone at ??:0
I20240501 07:59:23.995386 7140 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:23.995466 9665 raft_consensus.cc:2932] T 72937d60b2014cb490cbbb89c1082519 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:24.001343 6732 catalog_manager.cc:5484] T 72937d60b2014cb490cbbb89c1082519 P 49adb9e99822494196ed22044bfed81f reported cstate change: config changed from index 2 to 3, VOTER fa6964452854415a85ad72f8639768d6 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:24.006800 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 72937d60b2014cb490cbbb89c1082519 with delete_type TABLET_DATA_TOMBSTONED (TS fa6964452854415a85ad72f8639768d6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44266
I20240501 07:59:24.008450 23479 tablet_replica.cc:330] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: stopping tablet replica
I20240501 07:59:24.009922 23479 raft_consensus.cc:2229] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:24.010056 23479 raft_consensus.cc:2258] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:24.010560 23479 ts_tablet_manager.cc:1875] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:24.012528 23479 ts_tablet_manager.cc:1888] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 07:59:24.012578 23479 log.cc:1196] T 72937d60b2014cb490cbbb89c1082519 P fa6964452854415a85ad72f8639768d6: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/72937d60b2014cb490cbbb89c1082519
I20240501 07:59:24.013131 6723 catalog_manager.cc:4830] TS fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153): tablet 72937d60b2014cb490cbbb89c1082519 (table rebalancer_extra_table_0 [id=2242a163a8eb4225a501098c9528575f]) successfully deleted
I20240501 07:59:24.180208 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:24.181792 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:24.186254 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:24.195605 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.199678 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.206640 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:24.215473 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 07:59:24.302583 6815 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.302623 9207 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.302763 6815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.302763 9207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.303205 9602 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.303354 9602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.304172 7208 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.304299 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.305660 6946 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.305692 7077 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.305790 6946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.305816 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.306375 7075 connection.cc:441] server connection from 127.0.99.135:53675 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.99.135:53675 (request call id 544) could send its response
W20240501 07:59:24.306432 9339 connection.cc:535] client connection to 127.0.99.135:36013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.306548 9339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.308058 23545 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.303847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.129:52707)
0501 07:59:24.304076 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.304093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.304187 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.307524 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
W20240501 07:59:24.308197 6815 consensus_peers.cc:485] T 69ff85c89fa443539b28c0d0cc299cfd P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.308720 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.304906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.132:48745)
0501 07:59:24.305173 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.305189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.305279 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.308345 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
W20240501 07:59:24.308897 7208 consensus_peers.cc:485] T ab7dcdd38eff43fe9fff9081ee644c8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.308964 7208 consensus_peers.cc:485] T 4b683d8146a34fc6b74315f7db5d6890 P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309022 7208 consensus_peers.cc:485] T 22cb0ec763ff4941acbfbb007995241d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309068 7208 consensus_peers.cc:462] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:24.309111 7208 consensus_peers.cc:485] T 9051ba90bad3430fa2e128c88ec7b6f4 P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309165 7208 consensus_peers.cc:485] T 8c0f4ea5ccba462d9468e3f445569eaa P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309218 7208 consensus_peers.cc:485] T 2ba12580263f4bd883ad7e003ebd6c51 P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309432 23546 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.304383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.133:59407)
0501 07:59:24.304608 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.304622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.304701 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.309029 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
W20240501 07:59:24.309547 9207 consensus_peers.cc:485] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309607 9207 consensus_peers.cc:485] T 6eeffa33e44b45e8bdb1a71544e28250 P 49adb9e99822494196ed22044bfed81f -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309651 9207 consensus_peers.cc:485] T df79be5e913144a786b208e0a1674951 P 49adb9e99822494196ed22044bfed81f -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309695 9207 consensus_peers.cc:485] T e90ac95b538e419ea78e36e54ad8b4ac P 49adb9e99822494196ed22044bfed81f -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.309901 23549 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.305355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.136:37873)
0501 07:59:24.305532 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.305543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.305616 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.309499 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
W20240501 07:59:24.310092 23554 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.307155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.134:36395)
0501 07:59:24.307357 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.307373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.307512 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.309475 (+ 1963us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
W20240501 07:59:24.310189 7077 consensus_peers.cc:485] T f334b66a84b94c3ba15004253b4cb714 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310209 9339 consensus_peers.cc:485] T 79ab7f97eb2e439f8c974636b5791824 P fa6964452854415a85ad72f8639768d6 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310258 7077 consensus_peers.cc:485] T ef5cce31b78f402dae9530fc3079c28b P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310263 9339 consensus_peers.cc:485] T 6171adbaa9b4481988269d9a57220deb P fa6964452854415a85ad72f8639768d6 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310281 9602 consensus_peers.cc:485] T 4f3e4cd27e4545baa255a01023cf1d68 P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310307 7077 consensus_peers.cc:485] T f723d182d04d439da6ab5b811448cf2d P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310341 9602 consensus_peers.cc:485] T b56cc0ecf4014eb99ffebbf975be91ca P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310353 7077 consensus_peers.cc:485] T c80723cb0ce4446483a5f1754e066081 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310379 9602 consensus_peers.cc:485] T da18394593a24e61ba2f90356645621a P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310396 7077 consensus_peers.cc:485] T a1d72a9cc6a940b7b15535c8db19c312 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310416 9602 consensus_peers.cc:485] T d38445516d1e49d88ef9707246d70115 P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.310941 23552 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.306531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.99.130:34909)
0501 07:59:24.306767 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.306778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.306906 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.310576 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
W20240501 07:59:24.311079 6946 consensus_peers.cc:485] T 767acae4338b4b0dba53109bf86c13fc P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.312655 6815 consensus_peers.cc:485] T 4882718ff90c4d939917f4970a9acb9e P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.317873 6815 consensus_peers.cc:485] T 3839146ca1f0452499eb6b5083097d00 P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.324723 6815 consensus_peers.cc:485] T 3c13af71c56c49c89f4ba43be9956a50 P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.338104 6815 consensus_peers.cc:485] T e9e0b9c5ac80488897e07511c3006c7f P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.356809 9207 consensus_peers.cc:485] T 6a550051551640dcab9733948bb0966f P 49adb9e99822494196ed22044bfed81f -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.410470 9602 consensus_peers.cc:485] T 17bb409ae7134b07b8a809a9d1aabdd4 P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.416410 6815 consensus_peers.cc:485] T 07fdc1e114774fdb8c84adfdbf1d63f2 P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:24.458197 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.468356 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:24.470968 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.471822 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 07:59:24.476226 9602 consensus_peers.cc:485] T b40ef631801d42ec954b19f7323dc00a P 7fa3a88c25964445975bb47e0feed460 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:24.482764 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:24.491930 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 07:59:24.493866 7208 consensus_peers.cc:462] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:24.509547 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 07:59:24.514640 6815 consensus_peers.cc:485] T 75a5833b927e402aa86f218408592de2 P 1a9029bee6a144938f68b363e4cca162 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:24.521119 7077 consensus_peers.cc:485] T e1d646c521664e1995b56a40f0758e66 P fc946354b5d04503a56122e59fc320be -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Couldn't send request to peer ac6c8bc1b74b483791e262abee7862cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:24.571346 7271 consensus_queue.cc:237] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:24.572685 9266 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:24.572715 9665 raft_consensus.cc:1261] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Refusing update from remote peer 15ab96f6477b4291932409b80af07213: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:24.573104 22231 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:24.573314 22876 consensus_queue.cc:1035] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:24.574887 22876 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:24.575294 9665 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:24.578621 6730 catalog_manager.cc:5484] T 80152bb09e4e42a09e106cf8b58178e3 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: config changed from index 6 to 8, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:24.580991 9266 raft_consensus.cc:2932] T 80152bb09e4e42a09e106cf8b58178e3 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:24.586040 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet 80152bb09e4e42a09e106cf8b58178e3 with delete_type TABLET_DATA_TOMBSTONED (TS fc946354b5d04503a56122e59fc320be not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44470
I20240501 07:59:24.588434 23634 tablet_replica.cc:330] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: stopping tablet replica
I20240501 07:59:24.589501 23634 raft_consensus.cc:2229] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:24.589634 23634 raft_consensus.cc:2258] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:24.590116 23634 ts_tablet_manager.cc:1875] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:24.592480 23634 ts_tablet_manager.cc:1888] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 07:59:24.592576 23634 log.cc:1196] T 80152bb09e4e42a09e106cf8b58178e3 P fc946354b5d04503a56122e59fc320be: Deleting WAL directory at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/80152bb09e4e42a09e106cf8b58178e3
I20240501 07:59:24.593154 6722 catalog_manager.cc:4830] TS fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): tablet 80152bb09e4e42a09e106cf8b58178e3 (table rebalance_test_table_3 [id=24cc2994760543fab0b49d279826702d]) successfully deleted
I20240501 07:59:24.721633 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.723860 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:24.727423 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:24.729923 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.739187 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:24.739199 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:24.762745 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:24.926422 22418 raft_consensus.cc:482] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:24.926550 22418 raft_consensus.cc:504] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:24.927057 22418 leader_election.cc:288] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
W20240501 07:59:24.929046 9339 leader_election.cc:334] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:24.929251 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
W20240501 07:59:24.929602 9337 leader_election.cc:341] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Not found: Tablet not found: 0d818b9a4d9d48d9bab2b1c621db6828
I20240501 07:59:24.929663 9337 leader_election.cc:302] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:24.929956 22418 raft_consensus.cc:2726] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:24.935676 7250 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:24.939877 9250 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.942974 6857 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 07:59:24.947779 9644 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:24.949180 7121 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:24.965973 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:24.966403 6989 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 07:59:24.987298 23276 raft_consensus.cc:482] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:24.987440 23276 raft_consensus.cc:504] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:24.987857 23276 leader_election.cc:288] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:24.988951 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
W20240501 07:59:24.991406 6946 leader_election.cc:334] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:24.992329 6945 leader_election.cc:302] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2; no voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:24.992766 23276 raft_consensus.cc:2726] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.001770 23264 raft_consensus.cc:482] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.001901 23264 raft_consensus.cc:504] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.002249 23264 leader_election.cc:288] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.004002 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
W20240501 07:59:25.004945 9602 leader_election.cc:334] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.005012 9602 leader_election.cc:302] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:25.005239 23264 raft_consensus.cc:2726] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.007484 23473 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.007592 23473 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.008054 23473 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:25.008780 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:25.010758 23442 raft_consensus.cc:482] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.010866 23442 raft_consensus.cc:504] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.011212 23442 leader_election.cc:288] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.011926 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
W20240501 07:59:25.012189 9207 leader_election.cc:334] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.012253 9207 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49adb9e99822494196ed22044bfed81f; no voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf
W20240501 07:59:25.012868 7077 leader_election.cc:334] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.012938 7077 leader_election.cc:302] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.013127 23442 raft_consensus.cc:2726] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.013376 23473 raft_consensus.cc:2726] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.018883 23442 raft_consensus.cc:482] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.018982 23442 raft_consensus.cc:504] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.019378 23442 leader_election.cc:288] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.019986 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
W20240501 07:59:25.020511 7077 leader_election.cc:334] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.020593 7077 leader_election.cc:302] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.020728 23442 raft_consensus.cc:2726] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.043196 7009 consensus_queue.cc:237] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:59:25.044603 7208 consensus_peers.cc:462] T a9bd69f172f440a98debac09a97b1f8d P 15ab96f6477b4291932409b80af07213 -> Peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.045091 9403 raft_consensus.cc:1261] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:25.045118 9665 raft_consensus.cc:1261] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Refusing update from remote peer 9a3888360d4d4ace96b6b909c160bfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:25.045826 23276 consensus_queue.cc:1035] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:25.046453 23777 consensus_queue.cc:1035] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:25.046959 6944 consensus_peers.cc:485] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9324ffeaa4124027b586425e86bc6de4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:25.051383 23276 raft_consensus.cc:2932] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.053807 9403 raft_consensus.cc:2932] T 9324ffeaa4124027b586425e86bc6de4 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.054361 9665 raft_consensus.cc:2932] T 9324ffeaa4124027b586425e86bc6de4 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.056726 21884 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.056819 21884 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.056875 6734 catalog_manager.cc:5484] T 9324ffeaa4124027b586425e86bc6de4 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: config changed from index -1 to 2, VOTER fc946354b5d04503a56122e59fc320be (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.057222 21884 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:25.058041 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
W20240501 07:59:25.058441 9339 leader_election.cc:334] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.058518 9337 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.058653 21884 raft_consensus.cc:2726] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.086578 22876 raft_consensus.cc:482] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.086715 22876 raft_consensus.cc:504] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.087175 22876 leader_election.cc:288] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.087519 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
W20240501 07:59:25.088667 7208 leader_election.cc:334] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.088738 7208 leader_election.cc:302] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15ab96f6477b4291932409b80af07213; no voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.088874 22876 raft_consensus.cc:2726] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.097867 23296 raft_consensus.cc:482] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.097975 23296 raft_consensus.cc:504] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.098366 23296 leader_election.cc:288] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.098764 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
W20240501 07:59:25.099681 6815 leader_election.cc:334] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.099749 6815 leader_election.cc:302] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162; no voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.101415 23296 raft_consensus.cc:2726] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.107529 23276 raft_consensus.cc:482] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.107654 23276 raft_consensus.cc:504] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.108088 23276 leader_election.cc:288] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.108724 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.108893 7140 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
W20240501 07:59:25.109133 6946 leader_election.cc:334] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.109230 6944 leader_election.cc:302] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.109449 23276 raft_consensus.cc:2781] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.109506 23276 raft_consensus.cc:482] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.109552 23276 raft_consensus.cc:3037] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.110743 23276 raft_consensus.cc:504] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.111127 23276 leader_election.cc:288] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.111788 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f08fbb906443759009170c6b5ad068" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:25.111908 7140 raft_consensus.cc:3037] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.112874 7140 raft_consensus.cc:2454] T 80f08fbb906443759009170c6b5ad068 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:59:25.113232 6944 leader_election.cc:302] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.113770 23777 raft_consensus.cc:2781] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.113860 23777 raft_consensus.cc:686] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:25.114027 23777 consensus_queue.cc:237] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.116832 6728 catalog_manager.cc:5484] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
W20240501 07:59:25.117144 6946 leader_election.cc:334] T 80f08fbb906443759009170c6b5ad068 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.123636 23442 raft_consensus.cc:482] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.123768 23442 raft_consensus.cc:504] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:25.124238 23442 leader_election.cc:288] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.124620 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
W20240501 07:59:25.125734 7077 leader_election.cc:334] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.125823 7077 leader_election.cc:302] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.126015 23442 raft_consensus.cc:2726] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.128693 23264 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.128834 23264 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.129310 23264 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.129734 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
W20240501 07:59:25.130563 9602 leader_election.cc:334] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.130636 9602 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.130849 23264 raft_consensus.cc:2726] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.131781 23442 raft_consensus.cc:482] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.131884 23442 raft_consensus.cc:504] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.132251 23442 leader_election.cc:288] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.133359 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
W20240501 07:59:25.134064 7077 leader_election.cc:334] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.134135 7077 leader_election.cc:302] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc946354b5d04503a56122e59fc320be; no voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.134331 23442 raft_consensus.cc:2726] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.160054 22778 raft_consensus.cc:482] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.160172 22778 raft_consensus.cc:504] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.160245 23264 raft_consensus.cc:482] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.160349 23264 raft_consensus.cc:504] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.160583 22778 leader_election.cc:288] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.160741 23264 leader_election.cc:288] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.161053 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.161293 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:25.161427 7009 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:25.161926 9602 leader_election.cc:302] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.162058 22778 raft_consensus.cc:482] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.162127 22778 raft_consensus.cc:504] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.162413 23264 raft_consensus.cc:2781] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.162468 23264 raft_consensus.cc:482] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.162518 23264 raft_consensus.cc:3037] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.162444 9602 leader_election.cc:334] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.162559 9602 leader_election.cc:302] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:25.162451 22778 leader_election.cc:288] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
W20240501 07:59:25.162717 9602 leader_election.cc:334] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.162798 22778 raft_consensus.cc:2726] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.162973 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:25.163761 23264 raft_consensus.cc:504] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.164086 23264 leader_election.cc:288] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
W20240501 07:59:25.164316 9602 leader_election.cc:334] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.164394 9602 leader_election.cc:302] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460; no voters: 15ab96f6477b4291932409b80af07213, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.164399 21884 raft_consensus.cc:482] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.164510 21884 raft_consensus.cc:504] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.164894 21884 leader_election.cc:288] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.164921 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7686c82864eb49a89981252c40136528" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:25.165026 7009 raft_consensus.cc:3037] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.165395 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:25.165984 7009 raft_consensus.cc:2454] T 7686c82864eb49a89981252c40136528 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 2.
W20240501 07:59:25.167079 9339 leader_election.cc:334] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.167145 9339 leader_election.cc:302] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.167340 21884 raft_consensus.cc:2726] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.167515 21884 raft_consensus.cc:482] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.167570 21884 raft_consensus.cc:504] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.167901 21884 leader_election.cc:288] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:25.168368 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:25.169821 22876 raft_consensus.cc:482] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
W20240501 07:59:25.169876 9339 leader_election.cc:334] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.169931 9339 leader_election.cc:302] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.169912 22876 raft_consensus.cc:504] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.170122 21884 raft_consensus.cc:2726] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.170318 22876 leader_election.cc:288] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.170581 9602 leader_election.cc:302] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.170718 22778 raft_consensus.cc:2726] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.170840 22778 raft_consensus.cc:2781] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.170979 22778 raft_consensus.cc:686] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [term 2 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:25.171137 22778 consensus_queue.cc:237] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
W20240501 07:59:25.171811 7208 leader_election.cc:334] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.172703 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:25.172819 9665 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:25.173239 7207 leader_election.cc:302] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.173466 22876 raft_consensus.cc:2781] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.173517 22876 raft_consensus.cc:482] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.173563 22876 raft_consensus.cc:3037] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.174139 9602 leader_election.cc:334] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.174748 22876 raft_consensus.cc:504] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.175099 22876 leader_election.cc:288] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.175869 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2044f11ba34d6f9225781c0b8fdb51" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:25.175983 9665 raft_consensus.cc:3037] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.176326 7208 leader_election.cc:334] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.176703 6737 catalog_manager.cc:5484] T 7686c82864eb49a89981252c40136528 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:25.177095 9665 raft_consensus.cc:2454] T 4e2044f11ba34d6f9225781c0b8fdb51 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:25.177441 7207 leader_election.cc:302] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.177672 22876 raft_consensus.cc:2781] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.177774 22876 raft_consensus.cc:686] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:25.177902 22876 consensus_queue.cc:237] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.180899 6737 catalog_manager.cc:5484] T 4e2044f11ba34d6f9225781c0b8fdb51 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:25.186847 22231 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.186967 22231 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.187355 22231 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.188156 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.188292 9403 raft_consensus.cc:2454] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:25.188570 7207 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:25.188939 22231 raft_consensus.cc:2781] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.189004 22231 raft_consensus.cc:482] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.189055 22231 raft_consensus.cc:3037] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.190174 22231 raft_consensus.cc:504] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.190502 22231 leader_election.cc:288] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.190801 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f560797734bc458ea6d9c538d8bacf7d" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:25.190941 9403 raft_consensus.cc:3037] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.192102 9403 raft_consensus.cc:2454] T f560797734bc458ea6d9c538d8bacf7d P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:25.192566 7207 leader_election.cc:302] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
W20240501 07:59:25.192770 7208 leader_election.cc:334] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.193120 22231 raft_consensus.cc:2781] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.193209 22231 raft_consensus.cc:686] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:25.193341 22231 consensus_queue.cc:237] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
W20240501 07:59:25.194201 7208 leader_election.cc:334] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.196645 6729 catalog_manager.cc:5484] T f560797734bc458ea6d9c538d8bacf7d P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:25.212074 22579 raft_consensus.cc:482] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.212203 22579 raft_consensus.cc:504] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.212656 22579 leader_election.cc:288] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.212745 22579 raft_consensus.cc:482] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.212806 22579 raft_consensus.cc:504] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.213080 22579 leader_election.cc:288] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
W20240501 07:59:25.213802 6815 leader_election.cc:334] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.214141 23276 raft_consensus.cc:482] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.214236 23276 raft_consensus.cc:504] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
W20240501 07:59:25.214470 6815 leader_election.cc:334] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.214741 23276 leader_election.cc:288] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.214681 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:25.215183 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:25.215646 6814 leader_election.cc:302] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162; no voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.215853 22579 raft_consensus.cc:2726] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:25.216044 6946 leader_election.cc:334] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.216115 6946 leader_election.cc:302] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2; no voters: 1a9029bee6a144938f68b363e4cca162, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.216172 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.216312 9403 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:25.216583 6814 leader_election.cc:302] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.216862 23777 raft_consensus.cc:2726] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.217005 22579 raft_consensus.cc:2781] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.217051 22579 raft_consensus.cc:482] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.217087 22579 raft_consensus.cc:3037] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.218329 22579 raft_consensus.cc:504] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.218696 22579 leader_election.cc:288] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.219156 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bf756d31de478aa7cba1ef071c75aa" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:25.219285 9403 raft_consensus.cc:3037] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.220319 9403 raft_consensus.cc:2454] T e8bf756d31de478aa7cba1ef071c75aa P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
W20240501 07:59:25.220329 6815 leader_election.cc:334] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.220634 6814 leader_election.cc:302] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.220850 22579 raft_consensus.cc:2781] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.220937 22579 raft_consensus.cc:686] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:25.221073 22579 consensus_queue.cc:237] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.223775 6732 catalog_manager.cc:5484] T e8bf756d31de478aa7cba1ef071c75aa P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:25.230697 22579 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.230825 22579 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.231261 22579 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
W20240501 07:59:25.232610 6815 leader_election.cc:334] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.233095 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:25.233223 9665 raft_consensus.cc:2454] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
I20240501 07:59:25.233572 6814 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.234650 22579 raft_consensus.cc:2781] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.234750 22579 raft_consensus.cc:482] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.234786 22579 raft_consensus.cc:3037] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.235843 22579 raft_consensus.cc:504] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
W20240501 07:59:25.237079 6815 leader_election.cc:334] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.237540 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d606ace12402a86ab5ce85a98cd44" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:25.237649 9665 raft_consensus.cc:3037] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.238626 9665 raft_consensus.cc:2454] T 8c3d606ace12402a86ab5ce85a98cd44 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:25.239075 23296 raft_consensus.cc:482] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.239157 23296 raft_consensus.cc:504] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.239455 23296 leader_election.cc:288] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.240120 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:25.240243 7009 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 1.
W20240501 07:59:25.240700 6815 leader_election.cc:334] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.240844 6815 leader_election.cc:302] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.241030 23296 raft_consensus.cc:2781] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.241070 23296 raft_consensus.cc:482] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.241103 23296 raft_consensus.cc:3037] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.241989 23296 raft_consensus.cc:504] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.242311 23296 leader_election.cc:288] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.243278 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a926c0a1d4b44ab8467f2bbafd4f23c" candidate_uuid: "1a9029bee6a144938f68b363e4cca162" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:25.243413 7009 raft_consensus.cc:3037] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.244422 7009 raft_consensus.cc:2454] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9029bee6a144938f68b363e4cca162 in term 2.
I20240501 07:59:25.244994 6815 leader_election.cc:302] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.245252 23296 raft_consensus.cc:2781] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.245393 23296 raft_consensus.cc:686] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:25.245514 23296 consensus_queue.cc:237] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
W20240501 07:59:25.247699 6815 leader_election.cc:334] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.249220 6736 catalog_manager.cc:5484] T 0a926c0a1d4b44ab8467f2bbafd4f23c P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:25.252084 6814 leader_election.cc:302] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 7fa3a88c25964445975bb47e0feed460; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.252739 23825 raft_consensus.cc:2781] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.252982 23825 raft_consensus.cc:686] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [term 2 LEADER]: Becoming Leader. State: Replica: 1a9029bee6a144938f68b363e4cca162, State: Running, Role: LEADER
I20240501 07:59:25.253180 23825 consensus_queue.cc:237] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.254563 22579 leader_election.cc:288] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 [CANDIDATE]: Term 2 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.255851 6736 catalog_manager.cc:5484] T 8c3d606ace12402a86ab5ce85a98cd44 P 1a9029bee6a144938f68b363e4cca162 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 1a9029bee6a144938f68b363e4cca162 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1a9029bee6a144938f68b363e4cca162" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:25.262519 23442 raft_consensus.cc:482] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.262631 23442 raft_consensus.cc:504] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.263048 23442 leader_election.cc:288] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.263449 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:25.263594 9665 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:25.263871 7076 leader_election.cc:302] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:25.263998 7077 leader_election.cc:334] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.264030 23442 raft_consensus.cc:2781] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.264089 23442 raft_consensus.cc:482] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.264134 23442 raft_consensus.cc:3037] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.265363 23442 raft_consensus.cc:504] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.265720 23442 leader_election.cc:288] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.265982 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941785ea6d6d459ebdce786b805d2a7a" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:25.266095 9665 raft_consensus.cc:3037] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.267504 7077 leader_election.cc:334] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.267897 9665 raft_consensus.cc:2454] T 941785ea6d6d459ebdce786b805d2a7a P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:25.268162 7076 leader_election.cc:302] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.268410 23442 raft_consensus.cc:2781] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.268498 23442 raft_consensus.cc:686] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [term 2 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:25.268637 23442 consensus_queue.cc:237] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.269835 22418 raft_consensus.cc:482] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.269937 22418 raft_consensus.cc:504] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.270334 22418 leader_election.cc:288] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111)
I20240501 07:59:25.270678 23755 raft_consensus.cc:482] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.270789 23755 raft_consensus.cc:504] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:25.271203 23755 leader_election.cc:288] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.271198 6735 catalog_manager.cc:5484] T 941785ea6d6d459ebdce786b805d2a7a P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.271917 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.272024 7139 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:25.272337 9205 leader_election.cc:302] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:25.272544 9207 leader_election.cc:334] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.272600 23755 raft_consensus.cc:2781] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.272651 23755 raft_consensus.cc:482] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.272694 23755 raft_consensus.cc:3037] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.272753 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
W20240501 07:59:25.273787 9339 leader_election.cc:334] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.273788 23755 raft_consensus.cc:504] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:25.273839 9339 leader_election.cc:302] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: 7fa3a88c25964445975bb47e0feed460, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.273954 21884 raft_consensus.cc:2726] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.274143 23755 leader_election.cc:288] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.274746 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9192ffe3b65341cb9ad6918f1e435408" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:25.274855 7139 raft_consensus.cc:3037] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.275184 9207 leader_election.cc:334] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.275842 7139 raft_consensus.cc:2454] T 9192ffe3b65341cb9ad6918f1e435408 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 2.
I20240501 07:59:25.276146 9205 leader_election.cc:302] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.276320 23755 raft_consensus.cc:2781] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.276420 23755 raft_consensus.cc:686] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [term 2 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:25.276551 23755 consensus_queue.cc:237] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } }
I20240501 07:59:25.279191 6735 catalog_manager.cc:5484] T 9192ffe3b65341cb9ad6918f1e435408 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 2 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } }
I20240501 07:59:25.282490 23755 raft_consensus.cc:482] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.282603 23755 raft_consensus.cc:504] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.286477 23755 leader_election.cc:288] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.286931 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:25.287110 6878 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:25.287365 9206 leader_election.cc:302] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
W20240501 07:59:25.287442 9207 leader_election.cc:334] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.287631 23755 raft_consensus.cc:2781] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.287688 23755 raft_consensus.cc:482] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.287731 23755 raft_consensus.cc:3037] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.288527 23276 raft_consensus.cc:482] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.288625 23276 raft_consensus.cc:504] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.288914 23755 raft_consensus.cc:504] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.289023 23276 leader_election.cc:288] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.289273 23755 leader_election.cc:288] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.289877 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.290030 7139 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:25.290315 6944 leader_election.cc:302] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.290482 23276 raft_consensus.cc:2781] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.290539 23276 raft_consensus.cc:482] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.290594 23276 raft_consensus.cc:3037] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.290653 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4af3cf8f044e2be0f8e3195f8c084" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:25.290805 6878 raft_consensus.cc:3037] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.292021 6878 raft_consensus.cc:2454] T 81e4af3cf8f044e2be0f8e3195f8c084 P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 2.
I20240501 07:59:25.292627 9206 leader_election.cc:302] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
W20240501 07:59:25.293619 9207 leader_election.cc:334] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
W20240501 07:59:25.294032 6946 leader_election.cc:334] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.294265 23777 raft_consensus.cc:482] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.294360 23777 raft_consensus.cc:504] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.294715 23777 leader_election.cc:288] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.295492 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.295616 7139 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:25.295924 6944 leader_election.cc:302] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.296123 23777 raft_consensus.cc:2781] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.296173 23777 raft_consensus.cc:482] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.296212 23777 raft_consensus.cc:3037] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.296780 6946 leader_election.cc:334] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.296988 23755 raft_consensus.cc:2781] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.297060 23755 raft_consensus.cc:686] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [term 2 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:25.297178 23755 consensus_queue.cc:237] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.297348 23777 raft_consensus.cc:504] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.297681 23777 leader_election.cc:288] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers fc946354b5d04503a56122e59fc320be (127.0.99.131:37107), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.298094 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd704e855744f2b1e95d125f282e5f" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:25.298192 7139 raft_consensus.cc:3037] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.299258 7139 raft_consensus.cc:2454] T 76cd704e855744f2b1e95d125f282e5f P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:59:25.299944 6735 catalog_manager.cc:5484] T 81e4af3cf8f044e2be0f8e3195f8c084 P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 2 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:25.300781 6944 leader_election.cc:302] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.302268 23473 raft_consensus.cc:482] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.302352 23473 raft_consensus.cc:504] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.302742 23473 leader_election.cc:288] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.304174 9250 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 07:59:25.305999 9207 leader_election.cc:334] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.307446 9644 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:25.307907 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:25.308454 9207 leader_election.cc:302] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49adb9e99822494196ed22044bfed81f; no voters: 9a3888360d4d4ace96b6b909c160bfc2, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.308602 23473 raft_consensus.cc:2726] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.308928 23777 raft_consensus.cc:2781] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.309013 23777 raft_consensus.cc:686] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
W20240501 07:59:25.309108 6946 leader_election.cc:334] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.309154 23777 consensus_queue.cc:237] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.309917 7250 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:25.294229 23276 raft_consensus.cc:504] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.311401 23276 leader_election.cc:288] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.315317 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8921a8351e04daba09a8e7e4f683fd9" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be"
I20240501 07:59:25.315441 7139 raft_consensus.cc:3037] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.315559 6735 catalog_manager.cc:5484] T 76cd704e855744f2b1e95d125f282e5f P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:25.316257 9382 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 07:59:25.316553 7139 raft_consensus.cc:2454] T f8921a8351e04daba09a8e7e4f683fd9 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:59:25.323371 23442 raft_consensus.cc:482] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.323475 23442 raft_consensus.cc:504] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.323865 23442 leader_election.cc:288] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.324110 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460" is_pre_election: true
I20240501 07:59:25.324193 23755 raft_consensus.cc:482] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.324245 9665 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:25.324273 23755 raft_consensus.cc:504] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.324621 23755 leader_election.cc:288] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.324887 7076 leader_election.cc:302] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.325352 23442 raft_consensus.cc:2781] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.325410 23442 raft_consensus.cc:482] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.325364 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162" is_pre_election: true
I20240501 07:59:25.325448 23442 raft_consensus.cc:3037] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.325488 6878 raft_consensus.cc:2454] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 1.
I20240501 07:59:25.325765 9206 leader_election.cc:302] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:25.325915 23473 raft_consensus.cc:2781] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.325953 23473 raft_consensus.cc:482] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.325989 23473 raft_consensus.cc:3037] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.326660 23442 raft_consensus.cc:504] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.327085 23442 leader_election.cc:288] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Requested vote from peers 7fa3a88c25964445975bb47e0feed460 (127.0.99.136:35111), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.327170 23473 raft_consensus.cc:504] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.327505 23473 leader_election.cc:288] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209)
I20240501 07:59:25.327673 9666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037ad1dda6448f7a026fc685642d5c6" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7fa3a88c25964445975bb47e0feed460"
I20240501 07:59:25.327785 9666 raft_consensus.cc:3037] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.328080 23276 raft_consensus.cc:482] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.328181 23276 raft_consensus.cc:504] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.328226 6944 leader_election.cc:302] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:25.328449 7077 leader_election.cc:334] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.328593 23276 leader_election.cc:288] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.328687 23276 raft_consensus.cc:2781] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.328765 23276 raft_consensus.cc:686] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:25.328753 9666 raft_consensus.cc:2454] T 1037ad1dda6448f7a026fc685642d5c6 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:25.328897 23276 consensus_queue.cc:237] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.329049 7076 leader_election.cc:302] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.329373 23640 raft_consensus.cc:2781] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.329458 23640 raft_consensus.cc:686] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [term 2 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:25.329502 6878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f40e32ef7f4757b59aef57320d866b" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a9029bee6a144938f68b363e4cca162"
I20240501 07:59:25.329619 6878 raft_consensus.cc:3037] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.329608 23640 consensus_queue.cc:237] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.330716 6878 raft_consensus.cc:2454] T 51f40e32ef7f4757b59aef57320d866b P 1a9029bee6a144938f68b363e4cca162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 2.
W20240501 07:59:25.330993 9207 leader_election.cc:334] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.331069 9206 leader_election.cc:302] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9029bee6a144938f68b363e4cca162, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:25.332353 23473 raft_consensus.cc:2781] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.332439 23473 raft_consensus.cc:686] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [term 2 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:25.332577 23473 consensus_queue.cc:237] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } }
I20240501 07:59:25.332654 6733 catalog_manager.cc:5484] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
W20240501 07:59:25.335256 7077 leader_election.cc:334] T 1037ad1dda6448f7a026fc685642d5c6 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
W20240501 07:59:25.335458 9207 leader_election.cc:334] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.335510 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:25.335633 7271 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:25.336284 6944 leader_election.cc:302] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
W20240501 07:59:25.336603 6946 leader_election.cc:334] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
W20240501 07:59:25.336730 6946 leader_election.cc:334] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.336622 17277 raft_consensus.cc:2781] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.336925 17277 raft_consensus.cc:482] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.336964 17277 raft_consensus.cc:3037] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.338033 17277 raft_consensus.cc:504] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
I20240501 07:59:25.338356 17277 leader_election.cc:288] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.339321 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70edef5951b74998b393ca86586abf61" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:25.339453 7271 raft_consensus.cc:3037] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.340507 7271 raft_consensus.cc:2454] T 70edef5951b74998b393ca86586abf61 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:59:25.340917 6944 leader_election.cc:302] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.341118 23777 raft_consensus.cc:2781] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.341197 23777 raft_consensus.cc:686] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
I20240501 07:59:25.341331 23777 consensus_queue.cc:237] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } }
W20240501 07:59:25.342085 6946 leader_election.cc:334] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.342370 6733 catalog_manager.cc:5484] T 51f40e32ef7f4757b59aef57320d866b P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 2 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "1a9029bee6a144938f68b363e4cca162" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37209 } } }
I20240501 07:59:25.343747 7121 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 07:59:25.345600 23818 raft_consensus.cc:482] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.345692 23818 raft_consensus.cc:504] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.346026 23818 leader_election.cc:288] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:25.347465 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.347585 9403 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
I20240501 07:59:25.347832 9601 leader_election.cc:302] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:25.348094 23818 raft_consensus.cc:2781] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.348147 23818 raft_consensus.cc:482] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.348182 23818 raft_consensus.cc:3037] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.348726 6733 catalog_manager.cc:5484] T f8921a8351e04daba09a8e7e4f683fd9 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:25.349231 23818 raft_consensus.cc:504] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.349521 23818 leader_election.cc:288] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
W20240501 07:59:25.349860 9602 leader_election.cc:334] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.353228 23276 raft_consensus.cc:482] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.353319 23276 raft_consensus.cc:504] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.353685 23276 leader_election.cc:288] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.354077 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:25.354205 9266 raft_consensus.cc:2454] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 1.
I20240501 07:59:25.354473 6945 leader_election.cc:302] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.354624 23276 raft_consensus.cc:2781] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.354671 23276 raft_consensus.cc:482] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.354759 23276 raft_consensus.cc:3037] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.355374 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604cff2480b942be85044096cdf95e7f" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:25.355484 9403 raft_consensus.cc:3037] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.355775 6946 leader_election.cc:334] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.355886 23276 raft_consensus.cc:504] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.356432 9403 raft_consensus.cc:2454] T 604cff2480b942be85044096cdf95e7f P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 2.
I20240501 07:59:25.356473 23276 leader_election.cc:288] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.356508 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4ad48cbfb947f4aec462093006f6c7" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:25.354696 23777 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.356616 9266 raft_consensus.cc:3037] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.356612 23777 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.356725 9601 leader_election.cc:302] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:25.356909 23777 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.357067 23818 raft_consensus.cc:2781] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.357141 23818 raft_consensus.cc:686] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [term 2 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:25.357257 23818 consensus_queue.cc:237] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.357286 9271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:25.357636 9266 raft_consensus.cc:2454] T 8a4ad48cbfb947f4aec462093006f6c7 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3888360d4d4ace96b6b909c160bfc2 in term 2.
I20240501 07:59:25.358058 6945 leader_election.cc:302] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters:
I20240501 07:59:25.358472 23777 raft_consensus.cc:2781] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.358551 23777 raft_consensus.cc:686] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 LEADER]: Becoming Leader. State: Replica: 9a3888360d4d4ace96b6b909c160bfc2, State: Running, Role: LEADER
W20240501 07:59:25.358691 6946 leader_election.cc:334] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.358676 23777 consensus_queue.cc:237] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
W20240501 07:59:25.358799 6946 leader_election.cc:334] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.358840 6946 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2; no voters: 49adb9e99822494196ed22044bfed81f, ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.359091 17277 raft_consensus.cc:2726] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 07:59:25.359360 9602 leader_election.cc:334] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.360641 6735 catalog_manager.cc:5484] T 604cff2480b942be85044096cdf95e7f P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:25.364249 6735 catalog_manager.cc:5484] T 70edef5951b74998b393ca86586abf61 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } }
I20240501 07:59:25.364506 6735 catalog_manager.cc:5484] T 8a4ad48cbfb947f4aec462093006f6c7 P 9a3888360d4d4ace96b6b909c160bfc2 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "9a3888360d4d4ace96b6b909c160bfc2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:25.365058 6857 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:25.366963 6989 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 07:59:25.367373 22231 raft_consensus.cc:482] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.367470 22231 raft_consensus.cc:504] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.367837 22231 leader_election.cc:288] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
W20240501 07:59:25.369380 7208 leader_election.cc:334] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.372390 21884 raft_consensus.cc:482] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.372493 21884 raft_consensus.cc:504] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.372587 9403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.372854 21884 leader_election.cc:288] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.373518 7207 leader_election.cc:302] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15ab96f6477b4291932409b80af07213; no voters: ac6c8bc1b74b483791e262abee7862cf, fa6964452854415a85ad72f8639768d6
W20240501 07:59:25.373943 9339 leader_election.cc:334] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.376724 22231 raft_consensus.cc:2726] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.379112 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
I20240501 07:59:25.379521 9337 leader_election.cc:302] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:25.379750 21884 raft_consensus.cc:2726] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.386701 23276 raft_consensus.cc:482] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.386806 23276 raft_consensus.cc:504] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.387215 23276 leader_election.cc:288] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.387593 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "9a3888360d4d4ace96b6b909c160bfc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fc946354b5d04503a56122e59fc320be" is_pre_election: true
W20240501 07:59:25.389569 6946 leader_election.cc:334] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.389645 6946 leader_election.cc:302] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:25.389921 23777 raft_consensus.cc:2726] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:25.394654 23442 raft_consensus.cc:482] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.394760 23442 raft_consensus.cc:504] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.395226 23442 leader_election.cc:288] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.395753 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:25.395874 7009 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:25.396351 7077 leader_election.cc:302] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
I20240501 07:59:25.396509 23442 raft_consensus.cc:2781] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.396560 23442 raft_consensus.cc:482] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.396610 23442 raft_consensus.cc:3037] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.397622 23442 raft_consensus.cc:504] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.398618 23442 leader_election.cc:288] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.398903 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a81efefaf414a6091da247b0986c322" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:25.399029 7009 raft_consensus.cc:3037] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.399994 7009 raft_consensus.cc:2454] T 0a81efefaf414a6091da247b0986c322 P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
W20240501 07:59:25.400487 7077 leader_election.cc:334] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.400702 7077 leader_election.cc:302] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3888360d4d4ace96b6b909c160bfc2, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:25.401448 7077 leader_election.cc:334] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.401618 23442 raft_consensus.cc:2781] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.401690 23442 raft_consensus.cc:686] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [term 2 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:25.401826 23442 consensus_queue.cc:237] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.404325 6735 catalog_manager.cc:5484] T 0a81efefaf414a6091da247b0986c322 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.425474 23755 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.425596 23755 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.426005 23755 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.426553 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2" is_pre_election: true
I20240501 07:59:25.426690 7009 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 2.
W20240501 07:59:25.427263 9207 leader_election.cc:334] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.427424 9207 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.427615 23755 raft_consensus.cc:2781] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 07:59:25.427666 23755 raft_consensus.cc:482] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.427701 23755 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:25.428733 23755 raft_consensus.cc:504] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.429062 23755 leader_election.cc:288] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675)
I20240501 07:59:25.429600 7009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee29c58cdc6544e8933300ad5faecb3a" candidate_uuid: "49adb9e99822494196ed22044bfed81f" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "9a3888360d4d4ace96b6b909c160bfc2"
I20240501 07:59:25.429730 7009 raft_consensus.cc:3037] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:25.430083 9207 leader_election.cc:334] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.430781 7009 raft_consensus.cc:2454] T ee29c58cdc6544e8933300ad5faecb3a P 9a3888360d4d4ace96b6b909c160bfc2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49adb9e99822494196ed22044bfed81f in term 3.
I20240501 07:59:25.431104 9207 leader_election.cc:302] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49adb9e99822494196ed22044bfed81f, 9a3888360d4d4ace96b6b909c160bfc2; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.431294 23755 raft_consensus.cc:2781] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:25.431371 23755 raft_consensus.cc:686] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [term 3 LEADER]: Becoming Leader. State: Replica: 49adb9e99822494196ed22044bfed81f, State: Running, Role: LEADER
I20240501 07:59:25.431493 23755 consensus_queue.cc:237] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } }
I20240501 07:59:25.434028 23264 raft_consensus.cc:482] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.434131 23264 raft_consensus.cc:504] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.434551 23264 leader_election.cc:288] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.435528 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.435655 9401 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
W20240501 07:59:25.435974 9602 leader_election.cc:334] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.436570 6735 catalog_manager.cc:5484] T ee29c58cdc6544e8933300ad5faecb3a P 49adb9e99822494196ed22044bfed81f reported cstate change: term changed from 2 to 3, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 49adb9e99822494196ed22044bfed81f (127.0.99.133). New cstate: current_term: 3 leader_uuid: "49adb9e99822494196ed22044bfed81f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "9a3888360d4d4ace96b6b909c160bfc2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39675 } } }
I20240501 07:59:25.437613 9601 leader_election.cc:302] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.437810 23264 raft_consensus.cc:2781] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.437875 23264 raft_consensus.cc:482] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.437923 23264 raft_consensus.cc:3037] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.439097 23264 raft_consensus.cc:504] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.439436 23264 leader_election.cc:288] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Requested vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013)
I20240501 07:59:25.440143 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7603404d6814a3ea8085a7d53772cb7" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:25.440266 9401 raft_consensus.cc:3037] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.440374 9602 leader_election.cc:334] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.441251 9401 raft_consensus.cc:2454] T a7603404d6814a3ea8085a7d53772cb7 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 2.
I20240501 07:59:25.441524 9601 leader_election.cc:302] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.441725 23264 raft_consensus.cc:2781] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.441835 23264 raft_consensus.cc:686] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [term 2 LEADER]: Becoming Leader. State: Replica: 7fa3a88c25964445975bb47e0feed460, State: Running, Role: LEADER
I20240501 07:59:25.441965 23264 consensus_queue.cc:237] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } }
I20240501 07:59:25.444559 6735 catalog_manager.cc:5484] T a7603404d6814a3ea8085a7d53772cb7 P 7fa3a88c25964445975bb47e0feed460 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 7fa3a88c25964445975bb47e0feed460 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "7fa3a88c25964445975bb47e0feed460" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } }
I20240501 07:59:25.461462 23818 raft_consensus.cc:482] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.461588 23818 raft_consensus.cc:504] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7fa3a88c25964445975bb47e0feed460" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 35111 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.462102 23818 leader_election.cc:288] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153), ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fc946354b5d04503a56122e59fc320be (127.0.99.131:37107)
I20240501 07:59:25.462260 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d818b9a4d9d48d9bab2b1c621db6828" candidate_uuid: "7fa3a88c25964445975bb47e0feed460" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.462379 9401 raft_consensus.cc:2454] T 0d818b9a4d9d48d9bab2b1c621db6828 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa3a88c25964445975bb47e0feed460 in term 1.
W20240501 07:59:25.463876 9602 leader_election.cc:334] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
W20240501 07:59:25.464294 9600 leader_election.cc:341] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Not found: Tablet not found: 0d818b9a4d9d48d9bab2b1c621db6828
I20240501 07:59:25.464376 9600 leader_election.cc:302] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7fa3a88c25964445975bb47e0feed460, fa6964452854415a85ad72f8639768d6; no voters: ac6c8bc1b74b483791e262abee7862cf, fc946354b5d04503a56122e59fc320be
I20240501 07:59:25.464686 23818 raft_consensus.cc:2726] T 0d818b9a4d9d48d9bab2b1c621db6828 P 7fa3a88c25964445975bb47e0feed460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskO45e9d/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240501 07:58:39.838238 9719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.838557 9719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.842144 9719 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:40.325225 9719 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 07:58:40.678575 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server 49adb9e99822494196ed22044bfed81f to server 15ab96f6477b4291932409b80af07213
W20240501 07:58:40.678896 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server 7fa3a88c25964445975bb47e0feed460 to server fc946354b5d04503a56122e59fc320be
W20240501 07:58:40.679111 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d3a5debb62c4bb2a70cb35425c5d2ef: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.679414 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server 7fa3a88c25964445975bb47e0feed460 to server 9a3888360d4d4ace96b6b909c160bfc2
W20240501 07:58:40.679569 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbc94645af2547f2897d473147120dc0: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.679654 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a1568fcce22f4efe9d2a8d214ec6ac0a: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.679723 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24cc2994760543fab0b49d279826702d: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.679803 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 74fc7bb11ad34e64bf0ced56aefa12b7: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.679876 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ea54f4fba04447f2a7ba1c5bf0a4fb37: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.679960 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c70f634763df42c8a6cc730777f0fda3: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.680013 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4f30bc26da4c4582b723e1ab47586a8e: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680060 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e658efc8bfca4c699ff1c5680ba5381c: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680104 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0d147294927c496ab944eaa27b4eb7a9: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.680143 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server fa6964452854415a85ad72f8639768d6 to server 1a9029bee6a144938f68b363e4cca162
W20240501 07:58:40.680189 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6cb6a410f06449a1a9bb80e4542585f2: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680238 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dae68e56a81f491399cd850fbef7cd2e: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680289 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eec42e978d1745efa8c79678ef5d0123: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680351 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c9f4bcc6262f4ecca1f57daaf60609df: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680394 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a1568fcce22f4efe9d2a8d214ec6ac0a: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680442 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 27ab3d3d5f18475fa304dba5270c4187: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680505 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5bf16c03ff40418ab1a63483d52907a8: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.680562 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d3a5debb62c4bb2a70cb35425c5d2ef: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680594 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 74fc7bb11ad34e64bf0ced56aefa12b7: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680629 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 16a2fc64af344f04907b5e2ed2ff67f0: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.680691 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0b8dc1a66bc74b4fa1aba2497ffe6e43: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.680740 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e3f92d832c844583ae778357ccc93086: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.680786 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c70f634763df42c8a6cc730777f0fda3: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.680830 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4f30bc26da4c4582b723e1ab47586a8e: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:40.680871 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0d147294927c496ab944eaa27b4eb7a9: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.680917 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e658efc8bfca4c699ff1c5680ba5381c: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:40.680959 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6cb6a410f06449a1a9bb80e4542585f2: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:40.681023 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eec42e978d1745efa8c79678ef5d0123: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:40.681057 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 27ab3d3d5f18475fa304dba5270c4187: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server fa6964452854415a85ad72f8639768d6
I20240501 07:58:40.709842 9719 rebalancer_tool.cc:1074] tablet e05498632f0945c29f723a4613a26a04: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:40.758508 9719 rebalancer_tool.cc:1074] tablet de03fdd63bd347dea4eb6e5d410da68e: '15ab96f6477b4291932409b80af07213' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:40.793457 9719 rebalancer_tool.cc:1074] tablet 58aab13671d647e485f9be94283b5239: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:40.828752 9719 rebalancer_tool.cc:1074] tablet 1671c966902c4eb98b238c4419c525fe: '1a9029bee6a144938f68b363e4cca162' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:40.859345 9719 rebalancer_tool.cc:1074] tablet 33e711b7a9684137969ce2bbffae40c2: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:40.891803 9719 rebalancer_tool.cc:1074] tablet 0d3d410c640243848a478aa2605c1bf0: '15ab96f6477b4291932409b80af07213' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:40.918969 9719 rebalancer_tool.cc:1074] tablet 4882718ff90c4d939917f4970a9acb9e: 'fc946354b5d04503a56122e59fc320be' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:40.946583 9719 rebalancer_tool.cc:1074] tablet 1f5dd2edfd7b4e28b226299dc2a4fc04: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:40.974112 9719 rebalancer_tool.cc:1074] tablet c80723cb0ce4446483a5f1754e066081: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:41.003222 9719 rebalancer_tool.cc:1074] tablet 37e73deb03274bba82ceb67087208bdd: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:41.031697 9719 rebalancer_tool.cc:1074] tablet 43b82e035ecb4e23973a2163e15c1dac: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:41.063129 9719 rebalancer_tool.cc:1074] tablet ab7dcdd38eff43fe9fff9081ee644c8d: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:41.093506 9719 rebalancer_tool.cc:1074] tablet 00cac44dfd0f4d97a3056b590748ba58: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:41.135160 9719 rebalancer_tool.cc:1074] tablet 48964a6c57584c3eacc6dffac29d77a8: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:41.197464 9719 rebalancer_tool.cc:1074] tablet 2304f5e5d09a47b793cbb4f60b255525: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:41.264490 9719 rebalancer_tool.cc:1074] tablet f4b3a231d67e4092a94d3539e31e62f4: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:41.322667 9719 rebalancer_tool.cc:1074] tablet 80152bb09e4e42a09e106cf8b58178e3: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:41.361419 9719 rebalancer_tool.cc:1074] tablet f334b66a84b94c3ba15004253b4cb714: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:41.399534 9719 rebalancer_tool.cc:1074] tablet 651beb63e03d40bca88f01ebed416754: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:41.432246 9719 rebalancer_tool.cc:1074] tablet 4e7f613e16c245de83fde8c277b8d350: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:44.303989 9719 rebalancer_tool.cc:1123] tablet 48964a6c57584c3eacc6dffac29d77a8: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move is abandoned: Not found: 48964a6c57584c3eacc6dffac29d77a8: tablet not found
I20240501 07:58:44.306003 9719 rebalancer_tool.cc:1123] tablet e05498632f0945c29f723a4613a26a04: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move is abandoned: Not found: e05498632f0945c29f723a4613a26a04: tablet not found
I20240501 07:58:44.306733 9719 rebalancer_tool.cc:1123] tablet de03fdd63bd347dea4eb6e5d410da68e: '15ab96f6477b4291932409b80af07213' -> 'fa6964452854415a85ad72f8639768d6' move is abandoned: Not found: de03fdd63bd347dea4eb6e5d410da68e: tablet not found
I20240501 07:58:44.307505 9719 rebalancer_tool.cc:1123] tablet 1f5dd2edfd7b4e28b226299dc2a4fc04: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move is abandoned: Not found: 1f5dd2edfd7b4e28b226299dc2a4fc04: tablet not found
I20240501 07:58:44.518805 9719 rebalancer_tool.cc:1123] tablet 37e73deb03274bba82ceb67087208bdd: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move is abandoned: Not found: 37e73deb03274bba82ceb67087208bdd: tablet not found
I20240501 07:58:44.744283 9719 rebalancer_tool.cc:1123] tablet f4b3a231d67e4092a94d3539e31e62f4: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move is abandoned: Not found: f4b3a231d67e4092a94d3539e31e62f4: tablet not found
I20240501 07:58:45.204104 9719 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:58:45.452406 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24cc2994760543fab0b49d279826702d: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:45.452641 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e3f92d832c844583ae778357ccc93086: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:45.452737 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server 49adb9e99822494196ed22044bfed81f to server 1a9029bee6a144938f68b363e4cca162
W20240501 07:58:45.452808 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e7755a2edf143b9b232d3d0a15695ec: could not find any suitable replica to move from server fa6964452854415a85ad72f8639768d6 to server fc946354b5d04503a56122e59fc320be
W20240501 07:58:45.452854 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 16a2fc64af344f04907b5e2ed2ff67f0: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:45.452895 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b930b491bfc45b296e95dda90955ff6: could not find any suitable replica to move from server ac6c8bc1b74b483791e262abee7862cf to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:45.452955 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6cb6a410f06449a1a9bb80e4542585f2: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:45.452992 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eec42e978d1745efa8c79678ef5d0123: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:45.453058 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0d147294927c496ab944eaa27b4eb7a9: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:45.453121 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c9f4bcc6262f4ecca1f57daaf60609df: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:45.453198 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a1568fcce22f4efe9d2a8d214ec6ac0a: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:45.453301 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec8cd84c027549079e88fc5284f3749b: could not find any suitable replica to move from server 49adb9e99822494196ed22044bfed81f to server 15ab96f6477b4291932409b80af07213
W20240501 07:58:45.453364 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 74fc7bb11ad34e64bf0ced56aefa12b7: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:45.453419 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ea54f4fba04447f2a7ba1c5bf0a4fb37: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:45.453466 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e3f92d832c844583ae778357ccc93086: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 7fa3a88c25964445975bb47e0feed460
I20240501 07:58:45.482200 9719 rebalancer_tool.cc:1074] tablet 3839146ca1f0452499eb6b5083097d00: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:45.530248 9719 rebalancer_tool.cc:1074] tablet 8d4ea1beadd542a894f002ae78d3b37f: '7fa3a88c25964445975bb47e0feed460' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:58:45.563846 9719 rebalancer_tool.cc:1074] tablet ee29c58cdc6544e8933300ad5faecb3a: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:45.595657 9719 rebalancer_tool.cc:1074] tablet 3c5e1de6fc394fcb98c64165d74ca317: '7fa3a88c25964445975bb47e0feed460' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:45.626143 9719 rebalancer_tool.cc:1074] tablet 69ff85c89fa443539b28c0d0cc299cfd: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:45.654736 9719 rebalancer_tool.cc:1074] tablet 0c86818cabaf49678cfbcce3480f2645: '1a9029bee6a144938f68b363e4cca162' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:58:47.196866 9719 rebalancer_tool.cc:1123] tablet 0d3d410c640243848a478aa2605c1bf0: '15ab96f6477b4291932409b80af07213' -> 'fa6964452854415a85ad72f8639768d6' move is abandoned: Not found: 0d3d410c640243848a478aa2605c1bf0: tablet not found
I20240501 07:58:47.208290 9719 rebalancer_tool.cc:1136] tablet 00cac44dfd0f4d97a3056b590748ba58: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:47.214659 9719 rebalancer_tool.cc:1136] tablet f334b66a84b94c3ba15004253b4cb714: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:58:47.215617 9719 rebalancer_tool.cc:1123] tablet 33e711b7a9684137969ce2bbffae40c2: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move is abandoned: Not found: 33e711b7a9684137969ce2bbffae40c2: tablet not found
I20240501 07:58:47.221688 9719 rebalancer_tool.cc:1136] tablet 1671c966902c4eb98b238c4419c525fe: '1a9029bee6a144938f68b363e4cca162' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:47.641067 9719 rebalancer_tool.cc:1136] tablet 4882718ff90c4d939917f4970a9acb9e: 'fc946354b5d04503a56122e59fc320be' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:47.998832 9719 rebalancer_tool.cc:1136] tablet 80152bb09e4e42a09e106cf8b58178e3: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:48.004765 9719 rebalancer_tool.cc:1136] tablet 58aab13671d647e485f9be94283b5239: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:48.010351 9719 rebalancer_tool.cc:1136] tablet c80723cb0ce4446483a5f1754e066081: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:58:48.015811 9719 rebalancer_tool.cc:1136] tablet 43b82e035ecb4e23973a2163e15c1dac: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:58:48.022790 9719 rebalancer_tool.cc:1136] tablet ab7dcdd38eff43fe9fff9081ee644c8d: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:48.031579 9719 rebalancer_tool.cc:1136] tablet 2304f5e5d09a47b793cbb4f60b255525: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:58:48.058722 9719 rebalancer_tool.cc:1074] tablet 0a81efefaf414a6091da247b0986c322: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:58:48.086225 9719 rebalancer_tool.cc:1074] tablet f723d182d04d439da6ab5b811448cf2d: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:48.118216 9719 rebalancer_tool.cc:1074] tablet f8921a8351e04daba09a8e7e4f683fd9: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:58:48.158805 9719 rebalancer_tool.cc:1074] tablet 82c30f6292754d97a839700be93f8c07: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:48.188346 9719 rebalancer_tool.cc:1074] tablet 22cb0ec763ff4941acbfbb007995241d: 'fa6964452854415a85ad72f8639768d6' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:48.228542 9719 rebalancer_tool.cc:1074] tablet 4f3e4cd27e4545baa255a01023cf1d68: 'ac6c8bc1b74b483791e262abee7862cf' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:58:48.264204 9719 rebalancer_tool.cc:1074] tablet c2f299abad9e415283470b4b214fc62f: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:48.289467 9719 rebalancer_tool.cc:1074] tablet 2eec88312feb44feb70401fe29a3bb83: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:48.320564 9719 rebalancer_tool.cc:1074] tablet 1037ad1dda6448f7a026fc685642d5c6: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:58:51.446945 9719 rebalancer_tool.cc:1136] tablet 3839146ca1f0452499eb6b5083097d00: '15ab96f6477b4291932409b80af07213' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:58:51.452634 9719 rebalancer_tool.cc:1136] tablet 4e7f613e16c245de83fde8c277b8d350: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:51.458346 9719 rebalancer_tool.cc:1136] tablet 651beb63e03d40bca88f01ebed416754: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:58:52.080993 9719 rebalancer_tool.cc:1136] tablet 0c86818cabaf49678cfbcce3480f2645: '1a9029bee6a144938f68b363e4cca162' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:58:52.106734 9719 rebalancer_tool.cc:1074] tablet 8c0f4ea5ccba462d9468e3f445569eaa: '1a9029bee6a144938f68b363e4cca162' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:58:52.152104 9719 rebalancer_tool.cc:1074] tablet b6421aafe6b34136bd8e1853cdf50786: '15ab96f6477b4291932409b80af07213' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:52.183789 9719 rebalancer_tool.cc:1074] tablet 2d9379f9bf86403ebf1695263db9a603: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:54.133641 9719 rebalancer_tool.cc:1136] tablet 4f3e4cd27e4545baa255a01023cf1d68: 'ac6c8bc1b74b483791e262abee7862cf' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:58:54.139775 9719 rebalancer_tool.cc:1136] tablet 22cb0ec763ff4941acbfbb007995241d: 'fa6964452854415a85ad72f8639768d6' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:54.150321 9719 rebalancer_tool.cc:1136] tablet f8921a8351e04daba09a8e7e4f683fd9: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:58:54.156916 9719 rebalancer_tool.cc:1136] tablet f723d182d04d439da6ab5b811448cf2d: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:58:54.481245 9719 rebalancer_tool.cc:1136] tablet 8d4ea1beadd542a894f002ae78d3b37f: '7fa3a88c25964445975bb47e0feed460' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:58:54.487354 9719 rebalancer_tool.cc:1136] tablet 82c30f6292754d97a839700be93f8c07: '1a9029bee6a144938f68b363e4cca162' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:54.492548 9719 rebalancer_tool.cc:1136] tablet 3c5e1de6fc394fcb98c64165d74ca317: '7fa3a88c25964445975bb47e0feed460' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:58:54.497901 9719 rebalancer_tool.cc:1136] tablet 0a81efefaf414a6091da247b0986c322: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:58:54.503796 9719 rebalancer_tool.cc:1136] tablet ee29c58cdc6544e8933300ad5faecb3a: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:54.536907 9719 rebalancer_tool.cc:1074] tablet 64e235bbc64a4dca8028ebf89c2b7b5e: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:58:54.593155 9719 rebalancer_tool.cc:1074] tablet 93af29aad2ff498dbd8bccaba12f6a70: 'fc946354b5d04503a56122e59fc320be' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:54.619863 9719 rebalancer_tool.cc:1074] tablet c2fa7739592145d5b90d98fecdbe02b2: '9a3888360d4d4ace96b6b909c160bfc2' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:54.646178 9719 rebalancer_tool.cc:1074] tablet c72ca77c85c94540905f236a4fc2e300: '49adb9e99822494196ed22044bfed81f' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:58:54.674494 9719 rebalancer_tool.cc:1074] tablet ae59405a605c4b30a508002a9df30945: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:58:54.676868 9719 rebalancer_tool.cc:1084] tablet 2828bc2999bb4a6a84a9213486981e85: '9a3888360d4d4ace96b6b909c160bfc2' -> '7fa3a88c25964445975bb47e0feed460' move ignored: Not found: 2828bc2999bb4a6a84a9213486981e85: tablet not found
I20240501 07:58:54.676959 9719 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:58:55.027603 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f3dc8e0a03a4aa6b0b2cf05b15747b0: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server ac6c8bc1b74b483791e262abee7862cf
W20240501 07:58:55.027829 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f3dc8e0a03a4aa6b0b2cf05b15747b0: could not find any suitable replica to move from server 49adb9e99822494196ed22044bfed81f to server 9a3888360d4d4ace96b6b909c160bfc2
W20240501 07:58:55.028352 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f3dc8e0a03a4aa6b0b2cf05b15747b0: could not find any suitable replica to move from server 1a9029bee6a144938f68b363e4cca162 to server fc946354b5d04503a56122e59fc320be
W20240501 07:58:55.028435 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a1568fcce22f4efe9d2a8d214ec6ac0a: could not find any suitable replica to move from server fc946354b5d04503a56122e59fc320be to server fa6964452854415a85ad72f8639768d6
W20240501 07:58:55.028514 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c9f4bcc6262f4ecca1f57daaf60609df: could not find any suitable replica to move from server 9a3888360d4d4ace96b6b909c160bfc2 to server 49adb9e99822494196ed22044bfed81f
W20240501 07:58:55.028589 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed5bc81d8f96454b8c9f400c29fb9ea6: could not find any suitable replica to move from server ac6c8bc1b74b483791e262abee7862cf to server 15ab96f6477b4291932409b80af07213
W20240501 07:58:55.028651 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e7c0b41904474ad7a867dd5037af0309: could not find any suitable replica to move from server ac6c8bc1b74b483791e262abee7862cf to server 15ab96f6477b4291932409b80af07213
W20240501 07:58:55.028715 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2242a163a8eb4225a501098c9528575f: could not find any suitable replica to move from server 7fa3a88c25964445975bb47e0feed460 to server 1a9029bee6a144938f68b363e4cca162
W20240501 07:58:55.028801 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f3dc8e0a03a4aa6b0b2cf05b15747b0: could not find any suitable replica to move from server 15ab96f6477b4291932409b80af07213 to server 7fa3a88c25964445975bb47e0feed460
W20240501 07:58:55.028846 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 43ba3c91d89e4d8f84c14a8ba9138816: could not find any suitable replica to move from server ac6c8bc1b74b483791e262abee7862cf to server 9a3888360d4d4ace96b6b909c160bfc2
W20240501 07:58:55.028903 9719 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b24fd09cea5543ed803e43c9dacf4e06: could not find any suitable replica to move from server fa6964452854415a85ad72f8639768d6 to server fc946354b5d04503a56122e59fc320be
I20240501 07:58:55.075039 9719 rebalancer_tool.cc:1074] tablet f334b66a84b94c3ba15004253b4cb714: '1a9029bee6a144938f68b363e4cca162' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:58:55.117183 9719 rebalancer_tool.cc:1074] tablet d3333244e06a4ad8bd3ecc73cb356c90: 'ac6c8bc1b74b483791e262abee7862cf' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:58:55.155706 9719 rebalancer_tool.cc:1074] tablet c684b44a722143c4a3c44caad7cd84be: 'fa6964452854415a85ad72f8639768d6' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:58:55.190794 9719 rebalancer_tool.cc:1074] tablet 3c5e1de6fc394fcb98c64165d74ca317: 'ac6c8bc1b74b483791e262abee7862cf' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:58:55.243278 9719 rebalancer_tool.cc:1074] tablet 79ab7f97eb2e439f8c974636b5791824: '7fa3a88c25964445975bb47e0feed460' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:58:55.274137 9719 rebalancer_tool.cc:1074] tablet 6a550051551640dcab9733948bb0966f: '9a3888360d4d4ace96b6b909c160bfc2' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:58:55.303527 9719 rebalancer_tool.cc:1074] tablet 748b249371a4464f88aeb0c4c8558998: '49adb9e99822494196ed22044bfed81f' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:58:59.386204 9719 rebalancer_tool.cc:1136] tablet 1037ad1dda6448f7a026fc685642d5c6: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:58:59.923122 9719 rebalancer_tool.cc:1136] tablet 2eec88312feb44feb70401fe29a3bb83: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:58:59.929237 9719 rebalancer_tool.cc:1136] tablet b6421aafe6b34136bd8e1853cdf50786: '15ab96f6477b4291932409b80af07213' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:59.935386 9719 rebalancer_tool.cc:1136] tablet c2f299abad9e415283470b4b214fc62f: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:58:59.942154 9719 rebalancer_tool.cc:1136] tablet 69ff85c89fa443539b28c0d0cc299cfd: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:58:59.971378 9719 rebalancer_tool.cc:1074] tablet da18394593a24e61ba2f90356645621a: 'fc946354b5d04503a56122e59fc320be' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:58:59.999935 9719 rebalancer_tool.cc:1074] tablet 0a926c0a1d4b44ab8467f2bbafd4f23c: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:00.036202 9719 rebalancer_tool.cc:1074] tablet 8c3d606ace12402a86ab5ce85a98cd44: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:00.071403 9719 rebalancer_tool.cc:1074] tablet 4f3e4cd27e4545baa255a01023cf1d68: '49adb9e99822494196ed22044bfed81f' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:59:00.100981 9719 rebalancer_tool.cc:1074] tablet 28f7eaacb7694552a8fa2f98a5a0c868: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
W20240501 07:59:00.516409 16864 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-01 07:59:00 UTC
I20240501 07:59:03.088604 9719 rebalancer_tool.cc:1136] tablet c684b44a722143c4a3c44caad7cd84be: 'fa6964452854415a85ad72f8639768d6' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:03.095343 9719 rebalancer_tool.cc:1136] tablet f334b66a84b94c3ba15004253b4cb714: '1a9029bee6a144938f68b363e4cca162' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:03.101635 9719 rebalancer_tool.cc:1136] tablet ae59405a605c4b30a508002a9df30945: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:03.110620 9719 rebalancer_tool.cc:1136] tablet c72ca77c85c94540905f236a4fc2e300: '49adb9e99822494196ed22044bfed81f' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:03.117990 9719 rebalancer_tool.cc:1136] tablet c2fa7739592145d5b90d98fecdbe02b2: '9a3888360d4d4ace96b6b909c160bfc2' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:03.124478 9719 rebalancer_tool.cc:1136] tablet 93af29aad2ff498dbd8bccaba12f6a70: 'fc946354b5d04503a56122e59fc320be' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:03.130532 9719 rebalancer_tool.cc:1136] tablet 748b249371a4464f88aeb0c4c8558998: '49adb9e99822494196ed22044bfed81f' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:03.136837 9719 rebalancer_tool.cc:1136] tablet 64e235bbc64a4dca8028ebf89c2b7b5e: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:03.817721 9719 rebalancer_tool.cc:1074] tablet d38445516d1e49d88ef9707246d70115: '49adb9e99822494196ed22044bfed81f' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:03.860064 9719 rebalancer_tool.cc:1074] tablet 651beb63e03d40bca88f01ebed416754: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:59:03.894184 9719 rebalancer_tool.cc:1074] tablet a7b180ca614d4eaf812ae2cf7485b715: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:03.926430 9719 rebalancer_tool.cc:1074] tablet 2ba12580263f4bd883ad7e003ebd6c51: '1a9029bee6a144938f68b363e4cca162' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:59:03.960608 9719 rebalancer_tool.cc:1074] tablet 82c30f6292754d97a839700be93f8c07: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:59:03.994936 9719 rebalancer_tool.cc:1074] tablet c977cb9c5d1b41a29e7671218b917315: '15ab96f6477b4291932409b80af07213' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:59:04.031632 9719 rebalancer_tool.cc:1074] tablet 0c86818cabaf49678cfbcce3480f2645: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:04.082106 9719 rebalancer_tool.cc:1074] tablet c80723cb0ce4446483a5f1754e066081: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:59:06.832973 9719 rebalancer_tool.cc:1136] tablet da18394593a24e61ba2f90356645621a: 'fc946354b5d04503a56122e59fc320be' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:07.123533 9719 rebalancer_tool.cc:1136] tablet 0a926c0a1d4b44ab8467f2bbafd4f23c: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:07.129683 9719 rebalancer_tool.cc:1136] tablet 6a550051551640dcab9733948bb0966f: '9a3888360d4d4ace96b6b909c160bfc2' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:59:07.135866 9719 rebalancer_tool.cc:1136] tablet 79ab7f97eb2e439f8c974636b5791824: '7fa3a88c25964445975bb47e0feed460' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:07.480847 9719 rebalancer_tool.cc:1136] tablet 3c5e1de6fc394fcb98c64165d74ca317: 'ac6c8bc1b74b483791e262abee7862cf' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:07.489094 9719 rebalancer_tool.cc:1136] tablet d3333244e06a4ad8bd3ecc73cb356c90: 'ac6c8bc1b74b483791e262abee7862cf' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:07.823834 9719 rebalancer_tool.cc:1136] tablet 2d9379f9bf86403ebf1695263db9a603: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:59:07.831180 9719 rebalancer_tool.cc:1136] tablet 8c0f4ea5ccba462d9468e3f445569eaa: '1a9029bee6a144938f68b363e4cca162' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:07.861853 9719 rebalancer_tool.cc:1074] tablet 8a4ad48cbfb947f4aec462093006f6c7: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:07.940941 9719 rebalancer_tool.cc:1074] tablet d30d4d6ddf3a40989b10cf5c10ef7082: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:07.969877 9719 rebalancer_tool.cc:1074] tablet 010c12e2cc194cdb87ff9299a8e42a77: '7fa3a88c25964445975bb47e0feed460' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:59:08.006147 9719 rebalancer_tool.cc:1074] tablet 4e7f613e16c245de83fde8c277b8d350: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:59:08.046450 9719 rebalancer_tool.cc:1074] tablet 9051ba90bad3430fa2e128c88ec7b6f4: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:59:08.095329 9719 rebalancer_tool.cc:1074] tablet 81e4af3cf8f044e2be0f8e3195f8c084: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:08.128652 9719 rebalancer_tool.cc:1074] tablet f7c14004fdb24e96bacbf8589518f3b1: '7fa3a88c25964445975bb47e0feed460' -> '15ab96f6477b4291932409b80af07213' move scheduled
I20240501 07:59:08.186846 9719 rebalancer_tool.cc:1074] tablet ab7dcdd38eff43fe9fff9081ee644c8d: 'fc946354b5d04503a56122e59fc320be' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:59:11.533612 9719 rebalancer_tool.cc:1136] tablet a7b180ca614d4eaf812ae2cf7485b715: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:11.539829 9719 rebalancer_tool.cc:1136] tablet d38445516d1e49d88ef9707246d70115: '49adb9e99822494196ed22044bfed81f' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:11.548223 9719 rebalancer_tool.cc:1136] tablet 28f7eaacb7694552a8fa2f98a5a0c868: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:11.554507 9719 rebalancer_tool.cc:1136] tablet 4f3e4cd27e4545baa255a01023cf1d68: '49adb9e99822494196ed22044bfed81f' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:11.560528 9719 rebalancer_tool.cc:1136] tablet 8c3d606ace12402a86ab5ce85a98cd44: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:11.566720 9719 rebalancer_tool.cc:1136] tablet 651beb63e03d40bca88f01ebed416754: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:11.849085 9719 rebalancer_tool.cc:1136] tablet 82c30f6292754d97a839700be93f8c07: '9a3888360d4d4ace96b6b909c160bfc2' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:11.855516 9719 rebalancer_tool.cc:1136] tablet 2ba12580263f4bd883ad7e003ebd6c51: '1a9029bee6a144938f68b363e4cca162' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:11.883628 9719 rebalancer_tool.cc:1074] tablet 4b683d8146a34fc6b74315f7db5d6890: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:11.910466 9719 rebalancer_tool.cc:1074] tablet 80f08fbb906443759009170c6b5ad068: 'fa6964452854415a85ad72f8639768d6' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:11.947242 9719 rebalancer_tool.cc:1074] tablet 8123a9666b814a8eb93098bf850ebd72: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:11.982251 9719 rebalancer_tool.cc:1074] tablet a7603404d6814a3ea8085a7d53772cb7: '49adb9e99822494196ed22044bfed81f' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:59:12.010689 9719 rebalancer_tool.cc:1074] tablet 4882718ff90c4d939917f4970a9acb9e: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
I20240501 07:59:12.040666 9719 rebalancer_tool.cc:1074] tablet b40ef631801d42ec954b19f7323dc00a: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:12.077142 9719 rebalancer_tool.cc:1074] tablet 7614555b22a847a2824f48d92611edda: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:14.340705 9719 rebalancer_tool.cc:1136] tablet 010c12e2cc194cdb87ff9299a8e42a77: '7fa3a88c25964445975bb47e0feed460' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:14.636757 9719 rebalancer_tool.cc:1136] tablet 8a4ad48cbfb947f4aec462093006f6c7: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:14.901297 9719 rebalancer_tool.cc:1136] tablet c80723cb0ce4446483a5f1754e066081: '15ab96f6477b4291932409b80af07213' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:14.908812 9719 rebalancer_tool.cc:1136] tablet 81e4af3cf8f044e2be0f8e3195f8c084: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:14.915436 9719 rebalancer_tool.cc:1136] tablet 0c86818cabaf49678cfbcce3480f2645: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:14.924161 9719 rebalancer_tool.cc:1136] tablet c977cb9c5d1b41a29e7671218b917315: '15ab96f6477b4291932409b80af07213' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:15.199049 9719 rebalancer_tool.cc:1136] tablet 9051ba90bad3430fa2e128c88ec7b6f4: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:15.236302 9719 rebalancer_tool.cc:1074] tablet ee29c58cdc6544e8933300ad5faecb3a: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:15.266469 9719 rebalancer_tool.cc:1074] tablet 3a54e29c093e4be485bf1af1e9ea7a72: '49adb9e99822494196ed22044bfed81f' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:59:15.299990 9719 rebalancer_tool.cc:1074] tablet b78be08b353141469d49c7460393a051: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move scheduled
I20240501 07:59:15.329180 9719 rebalancer_tool.cc:1074] tablet 2b09efd2448a4be7b63636b66e56a030: 'ac6c8bc1b74b483791e262abee7862cf' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:15.373229 9719 rebalancer_tool.cc:1074] tablet 4aea3c6384b74a32a7605f9777534618: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:15.410404 9719 rebalancer_tool.cc:1074] tablet f8921a8351e04daba09a8e7e4f683fd9: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move scheduled
I20240501 07:59:17.843835 9719 rebalancer_tool.cc:1136] tablet 8123a9666b814a8eb93098bf850ebd72: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:17.850116 9719 rebalancer_tool.cc:1136] tablet 80f08fbb906443759009170c6b5ad068: 'fa6964452854415a85ad72f8639768d6' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:17.856176 9719 rebalancer_tool.cc:1136] tablet ab7dcdd38eff43fe9fff9081ee644c8d: 'fc946354b5d04503a56122e59fc320be' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:17.862376 9719 rebalancer_tool.cc:1136] tablet 4b683d8146a34fc6b74315f7db5d6890: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:17.868909 9719 rebalancer_tool.cc:1136] tablet f7c14004fdb24e96bacbf8589518f3b1: '7fa3a88c25964445975bb47e0feed460' -> '15ab96f6477b4291932409b80af07213' move completed: OK
I20240501 07:59:17.875101 9719 rebalancer_tool.cc:1136] tablet 4e7f613e16c245de83fde8c277b8d350: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:59:17.881192 9719 rebalancer_tool.cc:1136] tablet d30d4d6ddf3a40989b10cf5c10ef7082: 'ac6c8bc1b74b483791e262abee7862cf' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:17.886628 9719 rebalancer_tool.cc:1136] tablet 7614555b22a847a2824f48d92611edda: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:17.892622 9719 rebalancer_tool.cc:1136] tablet a7603404d6814a3ea8085a7d53772cb7: '49adb9e99822494196ed22044bfed81f' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:17.921798 9719 rebalancer_tool.cc:1074] tablet e11a47afed3a44afa75f0dd363098c22: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:59:17.954579 9719 rebalancer_tool.cc:1074] tablet 791b17b0049c4bbbbf56934dcfe2f742: 'ac6c8bc1b74b483791e262abee7862cf' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:17.986035 9719 rebalancer_tool.cc:1074] tablet c0cf86e678084bc3b948968e5af0365d: '7fa3a88c25964445975bb47e0feed460' -> '49adb9e99822494196ed22044bfed81f' move scheduled
I20240501 07:59:18.014495 9719 rebalancer_tool.cc:1074] tablet 0a2d5511dc0d4e8cb6d71ca894da4884: 'ac6c8bc1b74b483791e262abee7862cf' -> 'fa6964452854415a85ad72f8639768d6' move scheduled
I20240501 07:59:18.050596 9719 rebalancer_tool.cc:1074] tablet 80152bb09e4e42a09e106cf8b58178e3: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move scheduled
I20240501 07:59:18.088841 9719 rebalancer_tool.cc:1074] tablet a1d72a9cc6a940b7b15535c8db19c312: 'ac6c8bc1b74b483791e262abee7862cf' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:21.524294 9719 rebalancer_tool.cc:1136] tablet 4aea3c6384b74a32a7605f9777534618: '49adb9e99822494196ed22044bfed81f' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:21.541988 9719 rebalancer_tool.cc:1136] tablet 3a54e29c093e4be485bf1af1e9ea7a72: '49adb9e99822494196ed22044bfed81f' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:21.548027 9719 rebalancer_tool.cc:1136] tablet b40ef631801d42ec954b19f7323dc00a: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:21.560029 9719 rebalancer_tool.cc:1136] tablet ee29c58cdc6544e8933300ad5faecb3a: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:21.571875 9719 rebalancer_tool.cc:1136] tablet 4882718ff90c4d939917f4970a9acb9e: '9a3888360d4d4ace96b6b909c160bfc2' -> 'ac6c8bc1b74b483791e262abee7862cf' move completed: OK
I20240501 07:59:21.601329 9719 rebalancer_tool.cc:1074] tablet 72937d60b2014cb490cbbb89c1082519: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:23.085857 9719 rebalancer_tool.cc:1136] tablet 791b17b0049c4bbbbf56934dcfe2f742: 'ac6c8bc1b74b483791e262abee7862cf' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:23.091567 9719 rebalancer_tool.cc:1136] tablet e11a47afed3a44afa75f0dd363098c22: 'fc946354b5d04503a56122e59fc320be' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:59:23.096688 9719 rebalancer_tool.cc:1136] tablet c0cf86e678084bc3b948968e5af0365d: '7fa3a88c25964445975bb47e0feed460' -> '49adb9e99822494196ed22044bfed81f' move completed: OK
I20240501 07:59:23.101716 9719 rebalancer_tool.cc:1136] tablet f8921a8351e04daba09a8e7e4f683fd9: '7fa3a88c25964445975bb47e0feed460' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:23.107299 9719 rebalancer_tool.cc:1136] tablet 2b09efd2448a4be7b63636b66e56a030: 'ac6c8bc1b74b483791e262abee7862cf' -> '9a3888360d4d4ace96b6b909c160bfc2' move completed: OK
I20240501 07:59:23.429292 9719 rebalancer_tool.cc:1074] tablet 0d818b9a4d9d48d9bab2b1c621db6828: '7fa3a88c25964445975bb47e0feed460' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:23.457818 9719 rebalancer_tool.cc:1074] tablet a9bd69f172f440a98debac09a97b1f8d: 'fc946354b5d04503a56122e59fc320be' -> 'ac6c8bc1b74b483791e262abee7862cf' move scheduled
W20240501 07:59:24.308126 23492 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:24.166222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:36013 (local address 127.0.0.1:41108)
0501 07:59:24.168641 (+ 2419us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:24.168650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:24.168730 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:24.307548 (+138818us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2374}
I20240501 07:59:24.586875 9719 rebalancer_tool.cc:1123] tablet 0d818b9a4d9d48d9bab2b1c621db6828: '7fa3a88c25964445975bb47e0feed460' -> 'fc946354b5d04503a56122e59fc320be' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:24.596285 9719 rebalancer_tool.cc:1136] tablet 0a2d5511dc0d4e8cb6d71ca894da4884: 'ac6c8bc1b74b483791e262abee7862cf' -> 'fa6964452854415a85ad72f8639768d6' move completed: OK
I20240501 07:59:24.602015 9719 rebalancer_tool.cc:1136] tablet b78be08b353141469d49c7460393a051: 'fa6964452854415a85ad72f8639768d6' -> '1a9029bee6a144938f68b363e4cca162' move completed: OK
I20240501 07:59:24.800735 9719 rebalancer_tool.cc:1136] tablet 72937d60b2014cb490cbbb89c1082519: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move completed: OK
I20240501 07:59:25.021857 9719 rebalancer_tool.cc:1136] tablet 80152bb09e4e42a09e106cf8b58178e3: 'fc946354b5d04503a56122e59fc320be' -> '7fa3a88c25964445975bb47e0feed460' move completed: OK
I20240501 07:59:25.060551 9719 rebalancer_tool.cc:1074] tablet 9324ffeaa4124027b586425e86bc6de4: 'fa6964452854415a85ad72f8639768d6' -> 'fc946354b5d04503a56122e59fc320be' move scheduled
I20240501 07:59:25.062490 9719 rebalancer_tool.cc:1084] tablet 6a58872a302445299d0de85b4d1d852c: '7fa3a88c25964445975bb47e0feed460' -> 'fc946354b5d04503a56122e59fc320be' move ignored: Not found: 6a58872a302445299d0de85b4d1d852c: tablet not found
I20240501 07:59:25.062563 9719 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): unacceptable health status UNAVAILABLE
I20240501 07:59:25.473656 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 6797
I20240501 07:59:25.479681 22418 raft_consensus.cc:482] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.479792 22418 raft_consensus.cc:504] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.480173 22418 leader_election.cc:288] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
I20240501 07:59:25.480763 22231 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.480870 22231 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.481287 22231 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.481258 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213" is_pre_election: true
I20240501 07:59:25.481374 7271 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 1.
I20240501 07:59:25.481644 9337 leader_election.cc:302] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:25.481644 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f" is_pre_election: true
I20240501 07:59:25.481781 9266 raft_consensus.cc:2454] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 1.
I20240501 07:59:25.481827 22418 raft_consensus.cc:2781] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.481889 22418 raft_consensus.cc:482] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.481930 22418 raft_consensus.cc:3037] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:25.482045 9339 leader_election.cc:334] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.482134 7207 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters:
I20240501 07:59:25.482290 22231 raft_consensus.cc:2781] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.482360 22231 raft_consensus.cc:482] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.482417 22231 raft_consensus.cc:3037] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.482925 22418 raft_consensus.cc:504] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.483566 22418 leader_election.cc:288] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309)
W20240501 07:59:25.483676 7208 leader_election.cc:334] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.483716 22231 raft_consensus.cc:504] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.484113 22231 leader_election.cc:288] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), 49adb9e99822494196ed22044bfed81f (127.0.99.133:46797)
I20240501 07:59:25.484151 7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e079a65e864d8f8fc84761766a7420" candidate_uuid: "fa6964452854415a85ad72f8639768d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "15ab96f6477b4291932409b80af07213"
I20240501 07:59:25.484241 7271 raft_consensus.cc:3037] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.484434 23640 raft_consensus.cc:482] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.484530 23640 raft_consensus.cc:504] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
W20240501 07:59:25.484979 7208 leader_election.cc:334] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.485080 23640 leader_election.cc:288] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:25.485270 7271 raft_consensus.cc:2454] T 84e079a65e864d8f8fc84761766a7420 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6964452854415a85ad72f8639768d6 in term 2.
I20240501 07:59:25.485404 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6" is_pre_election: true
I20240501 07:59:25.485416 9266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4908fb61c5d4ffdbdc9a22cc3457dd2" candidate_uuid: "15ab96f6477b4291932409b80af07213" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49adb9e99822494196ed22044bfed81f"
I20240501 07:59:25.485522 9401 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 1.
I20240501 07:59:25.485538 9266 raft_consensus.cc:3037] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.485591 9337 leader_election.cc:302] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab96f6477b4291932409b80af07213, fa6964452854415a85ad72f8639768d6; no voters:
I20240501 07:59:25.485728 22418 raft_consensus.cc:2781] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.485833 22418 raft_consensus.cc:686] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [term 2 LEADER]: Becoming Leader. State: Replica: fa6964452854415a85ad72f8639768d6, State: Running, Role: LEADER
I20240501 07:59:25.485970 22418 consensus_queue.cc:237] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } }
I20240501 07:59:25.486719 9266 raft_consensus.cc:2454] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 49adb9e99822494196ed22044bfed81f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab96f6477b4291932409b80af07213 in term 2.
I20240501 07:59:25.487093 7207 leader_election.cc:302] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15ab96f6477b4291932409b80af07213, 49adb9e99822494196ed22044bfed81f; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.487270 7076 leader_election.cc:302] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters:
W20240501 07:59:25.487583 9206 connection.cc:535] client connection to 127.0.99.129:37209 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:25.488790 23546 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:25.488070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.129:37209 (local address 127.0.99.133:32889)
0501 07:59:25.488236 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:25.488248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:25.488329 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:25.488684 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37209: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98}
W20240501 07:59:25.488919 9206 consensus_peers.cc:485] T 03a798f72da143c3a448e918eb594445 P 49adb9e99822494196ed22044bfed81f -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37209: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.489117 9339 leader_election.cc:334] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
W20240501 07:59:25.489188 7077 leader_election.cc:334] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.489284 23640 raft_consensus.cc:2781] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:25.489339 23640 raft_consensus.cc:482] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac6c8bc1b74b483791e262abee7862cf)
I20240501 07:59:25.489391 23640 raft_consensus.cc:3037] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.490347 6735 catalog_manager.cc:5484] T 84e079a65e864d8f8fc84761766a7420 P fa6964452854415a85ad72f8639768d6 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fa6964452854415a85ad72f8639768d6 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "fa6964452854415a85ad72f8639768d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } }
I20240501 07:59:25.490514 23640 raft_consensus.cc:504] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.490871 23640 leader_election.cc:288] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Requested vote from peers ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013), fa6964452854415a85ad72f8639768d6 (127.0.99.134:42153)
I20240501 07:59:25.491379 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 6928
W20240501 07:59:25.492338 7077 leader_election.cc:334] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac6c8bc1b74b483791e262abee7862cf (127.0.99.135:36013): Network error: Client connection negotiation failed: client connection to 127.0.99.135:36013: connect: Connection refused (error 111)
I20240501 07:59:25.492667 22231 raft_consensus.cc:2781] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.492781 22231 raft_consensus.cc:686] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [term 2 LEADER]: Becoming Leader. State: Replica: 15ab96f6477b4291932409b80af07213, State: Running, Role: LEADER
I20240501 07:59:25.492935 22231 consensus_queue.cc:237] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } }
I20240501 07:59:25.493345 9401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aea3c6384b74a32a7605f9777534618" candidate_uuid: "fc946354b5d04503a56122e59fc320be" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fa6964452854415a85ad72f8639768d6"
I20240501 07:59:25.493477 9401 raft_consensus.cc:3037] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:25.494423 9401 raft_consensus.cc:2454] T 4aea3c6384b74a32a7605f9777534618 P fa6964452854415a85ad72f8639768d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc946354b5d04503a56122e59fc320be in term 2.
I20240501 07:59:25.496845 6733 catalog_manager.cc:5484] T a4908fb61c5d4ffdbdc9a22cc3457dd2 P 15ab96f6477b4291932409b80af07213 reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to 15ab96f6477b4291932409b80af07213 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "15ab96f6477b4291932409b80af07213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "49adb9e99822494196ed22044bfed81f" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46797 } } peers { permanent_uuid: "15ab96f6477b4291932409b80af07213" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41309 } } }
I20240501 07:59:25.497318 7076 leader_election.cc:302] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: fa6964452854415a85ad72f8639768d6, fc946354b5d04503a56122e59fc320be; no voters: ac6c8bc1b74b483791e262abee7862cf
I20240501 07:59:25.497517 23640 raft_consensus.cc:2781] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:25.497656 23640 raft_consensus.cc:686] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [term 2 LEADER]: Becoming Leader. State: Replica: fc946354b5d04503a56122e59fc320be, State: Running, Role: LEADER
I20240501 07:59:25.497831 23640 consensus_queue.cc:237] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } }
I20240501 07:59:25.501878 6733 catalog_manager.cc:5484] T 4aea3c6384b74a32a7605f9777534618 P fc946354b5d04503a56122e59fc320be reported cstate change: term changed from 1 to 2, leader changed from ac6c8bc1b74b483791e262abee7862cf (127.0.99.135) to fc946354b5d04503a56122e59fc320be (127.0.99.131). New cstate: current_term: 2 leader_uuid: "fc946354b5d04503a56122e59fc320be" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac6c8bc1b74b483791e262abee7862cf" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 36013 } } peers { permanent_uuid: "fa6964452854415a85ad72f8639768d6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42153 } } peers { permanent_uuid: "fc946354b5d04503a56122e59fc320be" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37107 } } }
I20240501 07:59:25.506724 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7059
W20240501 07:59:25.515996 9205 connection.cc:535] client connection to 127.0.99.131:37107 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:25.516388 9337 connection.cc:535] client connection to 127.0.99.131:37107 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 07:59:25.516813 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 7190
W20240501 07:59:25.516801 9205 consensus_peers.cc:485] T 78e78a3c7f504121aa5e63e345b1c86f P 49adb9e99822494196ed22044bfed81f -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:37107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.518664 9337 consensus_peers.cc:485] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:37107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.521986 9339 consensus_peers.cc:485] T 12f4efcd9e274be483079915323045ad P fa6964452854415a85ad72f8639768d6 -> Peer 9a3888360d4d4ace96b6b909c160bfc2 (127.0.99.130:39675): Couldn't send request to peer 9a3888360d4d4ace96b6b909c160bfc2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:39675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.522547 9205 consensus_peers.cc:485] T ae59405a605c4b30a508002a9df30945 P 49adb9e99822494196ed22044bfed81f -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:37107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.524288 9337 consensus_peers.cc:485] T c93872326cd24c539432897bbd29ed0a P fa6964452854415a85ad72f8639768d6 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:37107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.526156 9337 connection.cc:535] client connection to 127.0.99.132:41309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 07:59:25.526882 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 9189
W20240501 07:59:25.526894 9337 consensus_peers.cc:485] T d3333244e06a4ad8bd3ecc73cb356c90 P fa6964452854415a85ad72f8639768d6 -> Peer 15ab96f6477b4291932409b80af07213 (127.0.99.132:41309): Couldn't send request to peer 15ab96f6477b4291932409b80af07213. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.132:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.528621 9338 consensus_peers.cc:485] T 0a2d5511dc0d4e8cb6d71ca894da4884 P fa6964452854415a85ad72f8639768d6 -> Peer 1a9029bee6a144938f68b363e4cca162 (127.0.99.129:37209): Couldn't send request to peer 1a9029bee6a144938f68b363e4cca162. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37209: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:25.532554 9337 consensus_peers.cc:485] T 791b17b0049c4bbbbf56934dcfe2f742 P fa6964452854415a85ad72f8639768d6 -> Peer fc946354b5d04503a56122e59fc320be (127.0.99.131:37107): Couldn't send request to peer fc946354b5d04503a56122e59fc320be. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:37107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:25.535791 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 9320
I20240501 07:59:25.544723 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 9584
I20240501 07:59:25.552879 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 6706
2024-05-01T07:59:25Z chronyd exiting
I20240501 07:59:25.571113 398 test_util.cc:175] -----------------------------------------------
I20240501 07:59:25.571180 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714550271492631-398-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (62973 ms)
[----------] 1 test from DDLDuringRebalancingTest (62973 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240501 07:59:25.571872 398 test_util.cc:268] Using random seed: -578991332
I20240501 07:59:25.572590 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:25.572629 398 ts_itest-base.cc:113] --------------
I20240501 07:59:25.572641 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 07:59:25.572651 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:59:25.572661 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:25Z Disabled control of system clock
I20240501 07:59:25.580277 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46851
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:46851
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 07:59:25.684144 23864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:25.684401 23864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:25.684461 23864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:25.687489 23864 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:25.687554 23864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:25.687577 23864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:25.687595 23864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:25.687613 23864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:25.691195 23864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:46851
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:25.692041 23864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:25.693149 23864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:25.701481 23872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:25.702154 23870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:25.703116 23869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:25.704041 23864 server_base.cc:1034] running on GCE node
I20240501 07:59:25.704550 23864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:25.705451 23864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:25.706625 23864 hybrid_clock.cc:648] HybridClock initialized: now 1714550365706602 us; error 37 us; skew 500 ppm
I20240501 07:59:25.708642 23864 webserver.cc:460] Webserver started at http://127.0.99.190:33137/ using document root <none> and password file <none>
I20240501 07:59:25.709218 23864 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:25.709277 23864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:25.709467 23864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:25.711210 23864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "db85d6043aa64cc88228b9f8528c0fde"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.711705 23864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "db85d6043aa64cc88228b9f8528c0fde"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.715108 23864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:25.717273 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:25.718241 23864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:25.718354 23864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "db85d6043aa64cc88228b9f8528c0fde"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.718444 23864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:25.742672 23864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:25.743355 23864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:25.743516 23864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:25.750052 23864 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:46851
I20240501 07:59:25.750058 23930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:46851 every 8 connection(s)
I20240501 07:59:25.751075 23864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:25.753628 23931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:25.755221 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 23864
I20240501 07:59:25.755383 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:25.759563 23931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde: Bootstrap starting.
I20240501 07:59:25.761658 23931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:25.762362 23931 log.cc:824] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:25.764052 23931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde: No bootstrap required, opened a new log
I20240501 07:59:25.766536 23931 raft_consensus.cc:348] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } }
I20240501 07:59:25.766702 23931 raft_consensus.cc:374] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:25.766746 23931 raft_consensus.cc:729] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db85d6043aa64cc88228b9f8528c0fde, State: Initialized, Role: FOLLOWER
I20240501 07:59:25.767149 23931 consensus_queue.cc:260] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } }
I20240501 07:59:25.767261 23931 raft_consensus.cc:388] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:25.767320 23931 raft_consensus.cc:482] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:25.767418 23931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:25.768118 23931 raft_consensus.cc:504] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } }
I20240501 07:59:25.768414 23931 leader_election.cc:302] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db85d6043aa64cc88228b9f8528c0fde; no voters:
I20240501 07:59:25.768647 23931 leader_election.cc:288] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:25.768787 23936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:25.769001 23936 raft_consensus.cc:686] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [term 1 LEADER]: Becoming Leader. State: Replica: db85d6043aa64cc88228b9f8528c0fde, State: Running, Role: LEADER
I20240501 07:59:25.769289 23936 consensus_queue.cc:237] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } }
I20240501 07:59:25.769670 23931 sys_catalog.cc:564] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:25.770916 23938 sys_catalog.cc:455] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader db85d6043aa64cc88228b9f8528c0fde. Latest consensus state: current_term: 1 leader_uuid: "db85d6043aa64cc88228b9f8528c0fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } } }
I20240501 07:59:25.771103 23938 sys_catalog.cc:458] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:25.770925 23937 sys_catalog.cc:455] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db85d6043aa64cc88228b9f8528c0fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db85d6043aa64cc88228b9f8528c0fde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46851 } } }
I20240501 07:59:25.771251 23937 sys_catalog.cc:458] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:25.771499 23947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:25.774350 23947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:25.778963 23947 catalog_manager.cc:1347] Generated new cluster ID: 85990ab735904f37b9e4faae703d287a
I20240501 07:59:25.779045 23947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:25.791347 23947 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:25.792241 23947 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:25.797080 23947 catalog_manager.cc:5857] T 00000000000000000000000000000000 P db85d6043aa64cc88228b9f8528c0fde: Generated new TSK 0
I20240501 07:59:25.797720 23947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:25.801365 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:25.908308 23955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:25.908514 23955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:25.908558 23955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:25.911649 23955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:25.911713 23955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:25.911809 23955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:25.915484 23955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.23955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:25.916404 23955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:25.917502 23955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:25.926954 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:25.927764 23961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:25.928426 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:25.929440 23955 server_base.cc:1034] running on GCE node
I20240501 07:59:25.929841 23955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:25.930395 23955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:25.931545 23955 hybrid_clock.cc:648] HybridClock initialized: now 1714550365931530 us; error 28 us; skew 500 ppm
I20240501 07:59:25.933409 23955 webserver.cc:460] Webserver started at http://127.0.99.129:32889/ using document root <none> and password file <none>
I20240501 07:59:25.933945 23955 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:25.934010 23955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:25.934180 23955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:25.935847 23955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.936313 23955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.939692 23955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:25.941931 23969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:25.942951 23955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:25.943084 23955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
format_stamp: "Formatted at 2024-05-01 07:59:25 on dist-test-slave-qm51"
I20240501 07:59:25.943176 23955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:25.955611 23955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:25.956182 23955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:25.956338 23955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:25.956866 23955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:25.957731 23955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:25.957778 23955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:25.957836 23955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:25.957867 23955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:25.965755 23955 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37675
I20240501 07:59:25.965795 24082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37675 every 8 connection(s)
I20240501 07:59:25.966689 23955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:25.975451 24083 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:25.975711 24083 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:25.976157 24083 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:25.976958 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 23955
I20240501 07:59:25.977061 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:25.977579 23895 ts_manager.cc:194] Registered new tserver with Master: 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:25.978740 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:41675
I20240501 07:59:25.979192 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:26.084724 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.084923 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.084964 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.087875 24086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.087934 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.088053 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:26.091687 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:26.092494 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:26.093569 24086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:26.102419 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.103022 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.104039 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.104882 24086 server_base.cc:1034] running on GCE node
I20240501 07:59:26.105264 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:26.105830 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:26.107028 24086 hybrid_clock.cc:648] HybridClock initialized: now 1714550366106978 us; error 39 us; skew 500 ppm
I20240501 07:59:26.108994 24086 webserver.cc:460] Webserver started at http://127.0.99.130:41011/ using document root <none> and password file <none>
I20240501 07:59:26.109563 24086 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:26.109627 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:26.109815 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:26.111513 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3748d2c0aeb740f98f8c2caabad21474"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.112039 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3748d2c0aeb740f98f8c2caabad21474"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.115301 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:26.117450 24100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.118415 24086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:26.118522 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3748d2c0aeb740f98f8c2caabad21474"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.118623 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:26.128547 24086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:26.129084 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:26.129231 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:26.129767 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:26.130599 24086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:26.130647 24086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.130713 24086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:26.130743 24086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.139132 24086 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40707
I20240501 07:59:26.139169 24213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40707 every 8 connection(s)
I20240501 07:59:26.140056 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:26.145108 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24086
I20240501 07:59:26.145222 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:26.147599 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:26.149227 24214 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:26.149483 24214 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:26.149961 24214 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:26.150866 23895 ts_manager.cc:194] Registered new tserver with Master: 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:26.151437 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55739
W20240501 07:59:26.253238 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.253468 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.253540 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.256546 24217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.256611 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.256723 24217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:26.260210 24217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:26.261210 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:26.262403 24217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:26.271281 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.271631 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.272498 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.273811 24217 server_base.cc:1034] running on GCE node
I20240501 07:59:26.274178 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:26.274724 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:26.275903 24217 hybrid_clock.cc:648] HybridClock initialized: now 1714550366275896 us; error 34 us; skew 500 ppm
I20240501 07:59:26.277845 24217 webserver.cc:460] Webserver started at http://127.0.99.131:43697/ using document root <none> and password file <none>
I20240501 07:59:26.278390 24217 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:26.278452 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:26.278628 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:26.280308 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9d371a71c02a4432aca2fed33b2545d5"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.280799 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9d371a71c02a4432aca2fed33b2545d5"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.284256 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:26.286477 24231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.287571 24217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:59:26.287704 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9d371a71c02a4432aca2fed33b2545d5"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.287813 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:26.318837 24217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:26.319499 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:26.319676 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:26.320202 24217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:26.321085 24217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:26.321205 24217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.321317 24217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:26.321354 24217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.330578 24217 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43575
I20240501 07:59:26.330688 24344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43575 every 8 connection(s)
I20240501 07:59:26.331630 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:26.333698 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24217
I20240501 07:59:26.333807 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:26.336097 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:26.340898 24345 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:26.341110 24345 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:26.341542 24345 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:26.342590 23895 ts_manager.cc:194] Registered new tserver with Master: 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:26.343428 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:44337
W20240501 07:59:26.445119 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.445322 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.445363 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.448542 24348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.448613 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.448681 24348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:26.452297 24348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:26.453142 24348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:26.454249 24348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:26.463218 24356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.463549 24353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.464386 24354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.465595 24348 server_base.cc:1034] running on GCE node
I20240501 07:59:26.465986 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:26.466529 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:26.467692 24348 hybrid_clock.cc:648] HybridClock initialized: now 1714550366467672 us; error 34 us; skew 500 ppm
I20240501 07:59:26.469568 24348 webserver.cc:460] Webserver started at http://127.0.99.132:39299/ using document root <none> and password file <none>
I20240501 07:59:26.470131 24348 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:26.470188 24348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:26.470356 24348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:26.472079 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.472524 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.475773 24348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:26.477841 24362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.478813 24348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:26.478909 24348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.479029 24348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:26.497843 24348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:26.498425 24348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:26.498602 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:26.499173 24348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:26.499995 24348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:26.500058 24348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.500097 24348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:26.500130 24348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.508287 24348 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43159
I20240501 07:59:26.508309 24475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43159 every 8 connection(s)
I20240501 07:59:26.509213 24348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:26.511942 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24348
I20240501 07:59:26.512084 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:26.514568 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:26.518730 24476 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:26.518988 24476 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:26.519596 24476 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:26.520715 23895 ts_manager.cc:194] Registered new tserver with Master: 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:26.521263 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:42803
W20240501 07:59:26.626507 24479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.626751 24479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.626807 24479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.629938 24479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.630012 24479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.630123 24479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:26.633802 24479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:26.634812 24479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:26.636041 24479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:26.644688 24484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.645668 24485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.646824 24487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.648013 24479 server_base.cc:1034] running on GCE node
I20240501 07:59:26.648384 24479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:26.648931 24479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:26.650081 24479 hybrid_clock.cc:648] HybridClock initialized: now 1714550366650066 us; error 33 us; skew 500 ppm
I20240501 07:59:26.652024 24479 webserver.cc:460] Webserver started at http://127.0.99.133:37823/ using document root <none> and password file <none>
I20240501 07:59:26.652549 24479 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:26.652608 24479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:26.652777 24479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:26.654381 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fdaa8b9ba4e649359a29a8c224761c46"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.654855 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fdaa8b9ba4e649359a29a8c224761c46"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.657972 24479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:26.660145 24493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.661083 24479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:26.661183 24479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fdaa8b9ba4e649359a29a8c224761c46"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.661263 24479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:26.676285 24479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:26.676836 24479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:26.676982 24479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:26.677464 24479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:26.678287 24479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:26.678340 24479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.678396 24479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:26.678421 24479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.686704 24479 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42155
I20240501 07:59:26.686725 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42155 every 8 connection(s)
I20240501 07:59:26.687644 24479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:26.690416 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24479
I20240501 07:59:26.690555 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:26.693113 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:26.697275 24607 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:26.697489 24607 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:26.697944 24607 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:26.698930 23895 ts_manager.cc:194] Registered new tserver with Master: fdaa8b9ba4e649359a29a8c224761c46 (127.0.99.133:42155)
I20240501 07:59:26.699532 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41821
W20240501 07:59:26.801988 24610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.802194 24610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.802239 24610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.805254 24610 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.805347 24610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.805437 24610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:26.809067 24610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:26.809948 24610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:26.811146 24610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:26.819897 24616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:26.820809 24618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.821800 24610 server_base.cc:1034] running on GCE node
W20240501 07:59:26.822860 24615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:26.823391 24610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:26.823952 24610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:26.825129 24610 hybrid_clock.cc:648] HybridClock initialized: now 1714550366825106 us; error 39 us; skew 500 ppm
I20240501 07:59:26.827222 24610 webserver.cc:460] Webserver started at http://127.0.99.134:42529/ using document root <none> and password file <none>
I20240501 07:59:26.827773 24610 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:26.827829 24610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:26.828006 24610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:26.829661 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e85aa03a5269488a94c778f62e1b9f06"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.830137 24610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e85aa03a5269488a94c778f62e1b9f06"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.833529 24610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 07:59:26.835723 24624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.836722 24610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:26.836822 24610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e85aa03a5269488a94c778f62e1b9f06"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:26.836913 24610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:26.863883 24610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:26.864552 24610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:26.864703 24610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:26.865216 24610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:26.866111 24610 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:26.866155 24610 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.866201 24610 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:26.866231 24610 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:26.874732 24610 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37171
I20240501 07:59:26.874765 24737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37171 every 8 connection(s)
I20240501 07:59:26.875698 24610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:26.879182 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24610
I20240501 07:59:26.879305 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:59:26.881556 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:26.885059 24738 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:26.885282 24738 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:26.885747 24738 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:26.886675 23895 ts_manager.cc:194] Registered new tserver with Master: e85aa03a5269488a94c778f62e1b9f06 (127.0.99.134:37171)
I20240501 07:59:26.887260 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:60387
I20240501 07:59:26.981822 24083 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
W20240501 07:59:26.993590 24741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:26.993891 24741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:26.993965 24741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:26.998440 24741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:26.998507 24741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:26.998627 24741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:27.002614 24741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:27.003597 24741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:27.004714 24741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:27.013252 24749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:27.014189 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:27.015060 24747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:27.016297 24741 server_base.cc:1034] running on GCE node
I20240501 07:59:27.016785 24741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:27.017370 24741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:27.018527 24741 hybrid_clock.cc:648] HybridClock initialized: now 1714550367018508 us; error 33 us; skew 500 ppm
I20240501 07:59:27.020514 24741 webserver.cc:460] Webserver started at http://127.0.99.135:46723/ using document root <none> and password file <none>
I20240501 07:59:27.021051 24741 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:27.021128 24741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:27.021306 24741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:27.023190 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5069f7538b3145fbaf4ccde56e0f3599"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.023641 24741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5069f7538b3145fbaf4ccde56e0f3599"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.026995 24741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:27.029304 24755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.030368 24741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:27.030478 24741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5069f7538b3145fbaf4ccde56e0f3599"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.030568 24741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:27.054665 24741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:27.055358 24741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:27.055531 24741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:27.056093 24741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:27.056988 24741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:27.057046 24741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.057107 24741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:27.057133 24741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.065119 24741 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41365
I20240501 07:59:27.065145 24868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41365 every 8 connection(s)
I20240501 07:59:27.066066 24741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:27.067859 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24741
I20240501 07:59:27.067978 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:59:27.070798 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:27.075907 24869 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:27.076165 24869 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:27.076771 24869 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:27.077855 23895 ts_manager.cc:194] Registered new tserver with Master: 5069f7538b3145fbaf4ccde56e0f3599 (127.0.99.135:41365)
I20240501 07:59:27.078400 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:57883
I20240501 07:59:27.153304 24214 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
W20240501 07:59:27.180542 24872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:27.180769 24872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:27.180840 24872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:27.183945 24872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:27.184021 24872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:27.184115 24872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:27.187700 24872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.24872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:27.188616 24872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:27.189749 24872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:27.199075 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:27.199613 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:27.200626 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:27.201552 24872 server_base.cc:1034] running on GCE node
I20240501 07:59:27.201954 24872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:27.202522 24872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:27.203666 24872 hybrid_clock.cc:648] HybridClock initialized: now 1714550367203651 us; error 37 us; skew 500 ppm
I20240501 07:59:27.205513 24872 webserver.cc:460] Webserver started at http://127.0.99.136:34649/ using document root <none> and password file <none>
I20240501 07:59:27.206059 24872 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:27.206122 24872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:27.206310 24872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:27.207948 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d6ea16b90d4446f4a2f9b4bf61c400fc"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.208395 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d6ea16b90d4446f4a2f9b4bf61c400fc"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.211757 24872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:27.213922 24886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.214913 24872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:27.215035 24872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d6ea16b90d4446f4a2f9b4bf61c400fc"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:27.215119 24872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:27.227732 24872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:27.228260 24872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:27.228407 24872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:27.228893 24872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:27.229703 24872 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:27.229750 24872 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.229809 24872 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:27.229835 24872 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:27.238055 24872 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35905
I20240501 07:59:27.238088 24999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35905 every 8 connection(s)
I20240501 07:59:27.238914 24872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:27.246994 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 24872
I20240501 07:59:27.247166 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:59:27.250907 25000 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:27.251147 25000 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:27.251591 25000 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:27.252559 23895 ts_manager.cc:194] Registered new tserver with Master: d6ea16b90d4446f4a2f9b4bf61c400fc (127.0.99.136:35905)
I20240501 07:59:27.253093 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:43289
I20240501 07:59:27.261375 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:59:27.270910 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:59:27.270990 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 24479
I20240501 07:59:27.277833 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 24610
I20240501 07:59:27.283360 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 24741
I20240501 07:59:27.288642 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO45e9d/build/debug/bin/kudu with pid 24872
I20240501 07:59:27.345358 24345 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:27.523783 24476 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:31.046011 23895 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:31.065966 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 48bd1d3faec840c5baf0454c33f8130f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.065894 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 48bd1d3faec840c5baf0454c33f8130f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.066004 24409 tablet_service.cc:1467] Processing CreateTablet for tablet e63f2991bfdf4e4a9aec4534605cf82e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.066663 24015 tablet_service.cc:1467] Processing CreateTablet for tablet e63f2991bfdf4e4a9aec4534605cf82e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.066838 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bd1d3faec840c5baf0454c33f8130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.066838 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bd1d3faec840c5baf0454c33f8130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.066864 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63f2991bfdf4e4a9aec4534605cf82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.067135 24016 tablet_service.cc:1467] Processing CreateTablet for tablet 133202f01ac942f9af1fe2d0229f38b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.067354 24016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133202f01ac942f9af1fe2d0229f38b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.068039 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63f2991bfdf4e4a9aec4534605cf82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.068516 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 133202f01ac942f9af1fe2d0229f38b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.068765 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133202f01ac942f9af1fe2d0229f38b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.066004 24410 tablet_service.cc:1467] Processing CreateTablet for tablet 48bd1d3faec840c5baf0454c33f8130f (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.070238 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bd1d3faec840c5baf0454c33f8130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.073813 25022 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:31.074472 25020 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:31.076189 25022 tablet_bootstrap.cc:654] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.076972 25022 log.cc:824] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:31.077021 25021 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:31.077148 25020 tablet_bootstrap.cc:654] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.077791 25020 log.cc:824] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:31.077832 24279 tablet_service.cc:1467] Processing CreateTablet for tablet 133202f01ac942f9af1fe2d0229f38b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.078750 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133202f01ac942f9af1fe2d0229f38b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.079121 25021 tablet_bootstrap.cc:654] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.079226 25020 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:31.079181 24277 tablet_service.cc:1467] Processing CreateTablet for tablet e63f2991bfdf4e4a9aec4534605cf82e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b7854abcc9e2456d9f07ab510573bdd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.079381 25020 ts_tablet_manager.cc:1367] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:31.079419 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63f2991bfdf4e4a9aec4534605cf82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.079814 25021 log.cc:824] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:31.081395 25021 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:31.081545 25022 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:31.081574 25021 ts_tablet_manager.cc:1367] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 07:59:31.081687 25022 ts_tablet_manager.cc:1367] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 07:59:31.083335 25020 raft_consensus.cc:348] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.083597 25020 raft_consensus.cc:374] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.083675 25020 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.084177 25020 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.085050 25020 ts_tablet_manager.cc:1398] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:31.085340 25022 raft_consensus.cc:348] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.085529 25022 raft_consensus.cc:374] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.085577 25022 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.085947 25022 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.086252 25021 raft_consensus.cc:348] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.086431 25021 raft_consensus.cc:374] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.086491 25021 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.086712 25022 ts_tablet_manager.cc:1398] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:31.086859 25021 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.087384 25022 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:31.087647 25021 ts_tablet_manager.cc:1398] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:31.087970 25021 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:31.088462 25022 tablet_bootstrap.cc:654] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.088915 25021 tablet_bootstrap.cc:654] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.089483 25020 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:31.089640 25022 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:31.089706 25022 ts_tablet_manager.cc:1367] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.089730 25027 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:31.090215 25022 raft_consensus.cc:348] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.090310 25022 raft_consensus.cc:374] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.090354 25022 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.090471 25022 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.090780 25020 tablet_bootstrap.cc:654] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.090823 25022 ts_tablet_manager.cc:1398] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.091890 25020 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:31.091966 25020 ts_tablet_manager.cc:1367] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:31.091995 25027 tablet_bootstrap.cc:654] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.092471 25020 raft_consensus.cc:348] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.092573 25020 raft_consensus.cc:374] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.092609 25020 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.092736 25020 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.092841 25027 log.cc:824] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:31.093096 25020 ts_tablet_manager.cc:1398] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.094449 25021 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:31.094537 25021 ts_tablet_manager.cc:1367] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:31.094751 25027 tablet_bootstrap.cc:492] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:31.094923 25027 ts_tablet_manager.cc:1367] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:31.094995 25021 raft_consensus.cc:348] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.095155 25021 raft_consensus.cc:374] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.095189 25021 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.095309 25021 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.095593 25021 ts_tablet_manager.cc:1398] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.095523 25020 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:31.097545 25020 tablet_bootstrap.cc:654] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.098543 25020 tablet_bootstrap.cc:492] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:31.098618 25020 ts_tablet_manager.cc:1367] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:31.098661 25027 raft_consensus.cc:348] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.098843 25027 raft_consensus.cc:374] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.098903 25027 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.099035 25020 raft_consensus.cc:348] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.099117 25020 raft_consensus.cc:374] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.099162 25020 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.099304 25020 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.099347 25027 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.099537 25020 ts_tablet_manager.cc:1398] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:31.100051 25027 ts_tablet_manager.cc:1398] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:31.100288 25027 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:31.101094 25027 tablet_bootstrap.cc:654] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.102200 25027 tablet_bootstrap.cc:492] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:31.102284 25027 ts_tablet_manager.cc:1367] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.102674 25027 raft_consensus.cc:348] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.102792 25027 raft_consensus.cc:374] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.102831 25027 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.102955 25027 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.103199 25027 ts_tablet_manager.cc:1398] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:59:31.144554 24215 tablet.cc:2369] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:31.146543 25037 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.146735 25037 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.147989 25037 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:31.152527 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:31.152529 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:31.152801 24037 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.152801 24430 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.153244 24233 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.153474 25037 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.153550 25037 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.153580 25037 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.154505 25037 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.154824 25037 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:31.155154 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:31.155165 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:31.155261 24037 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.155273 24430 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.156105 24430 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:31.156404 24233 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.156538 24037 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:31.156603 25037 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.156797 25037 raft_consensus.cc:686] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:31.157099 25037 consensus_queue.cc:237] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:31.159739 23894 catalog_manager.cc:5484] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:31.206420 25034 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.206584 25034 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.207724 25034 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.211552 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:31.211678 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:31.211776 24299 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:31.211934 24168 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:31.212213 23973 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.212467 25034 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.212528 25034 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.212571 25034 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.213361 25034 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.213661 25034 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.213968 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:31.213991 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:31.214071 24299 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.214090 24168 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.214918 24168 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:31.215227 23970 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:31.215369 25034 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.215420 24299 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:31.215447 25034 raft_consensus.cc:686] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:31.215770 25034 consensus_queue.cc:237] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.218420 23894 catalog_manager.cc:5484] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:31.221158 24084 tablet.cc:2369] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:31.234730 25029 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.234845 25029 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.235807 25029 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:31.239998 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:31.240000 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:31.240130 24037 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:31.240130 24430 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:31.240809 24101 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:31.241034 25029 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.241093 25029 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.241135 25029 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.242136 25029 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.242447 25029 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:31.242765 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:31.242874 24037 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.242856 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:31.242985 24430 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.243692 24037 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:31.243840 24430 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:31.243976 24101 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:31.244145 25029 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.244349 25029 raft_consensus.cc:686] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:31.244653 25029 consensus_queue.cc:237] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.247380 23894 catalog_manager.cc:5484] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:31.263198 24477 tablet.cc:2369] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:31.336422 24346 tablet.cc:2369] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:31.397390 25050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:31.397678 25050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:31.400833 25050 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:59:31.485479 24430 raft_consensus.cc:1261] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.486090 24168 raft_consensus.cc:1261] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.486141 25029 consensus_queue.cc:1035] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.486302 24299 raft_consensus.cc:1261] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.486999 25026 consensus_queue.cc:1035] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.487470 25026 consensus_queue.cc:1035] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.488490 24037 raft_consensus.cc:1261] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.490866 25029 consensus_queue.cc:1035] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.494863 24036 raft_consensus.cc:1261] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.495467 24428 raft_consensus.cc:1261] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:31.495985 25041 consensus_queue.cc:1035] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.499085 25037 consensus_queue.cc:1035] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:31.503235 25078 mvcc.cc:205] Tried to move back new op lower bound from 7022798321597149184 to 7022798320620490752. Current Snapshot: MvccSnapshot[applied={T|T < 7022798321597149184}]
I20240501 07:59:31.506282 25079 mvcc.cc:205] Tried to move back new op lower bound from 7022798321634095104 to 7022798320262074368. Current Snapshot: MvccSnapshot[applied={T|T < 7022798321634095104}]
I20240501 07:59:31.513659 25076 mvcc.cc:205] Tried to move back new op lower bound from 7022798321634095104 to 7022798320262074368. Current Snapshot: MvccSnapshot[applied={T|T < 7022798321634095104}]
I20240501 07:59:31.518329 25083 mvcc.cc:205] Tried to move back new op lower bound from 7022798321597149184 to 7022798320620490752. Current Snapshot: MvccSnapshot[applied={T|T < 7022798321597149184}]
INSERT report
rows total: 2000
time total: 118.118 ms
time per row: 0.0590591 ms
I20240501 07:59:31.564708 23891 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:31.569777 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb726fc00544504a34a37bf3226f889 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.569916 24279 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb726fc00544504a34a37bf3226f889 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.570118 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb726fc00544504a34a37bf3226f889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.570255 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb726fc00544504a34a37bf3226f889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.570855 24278 tablet_service.cc:1467] Processing CreateTablet for tablet c82547c4a0a8475b9e9a3aaf5a486438 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.571116 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82547c4a0a8475b9e9a3aaf5a486438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.571619 24015 tablet_service.cc:1467] Processing CreateTablet for tablet c82547c4a0a8475b9e9a3aaf5a486438 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.571879 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82547c4a0a8475b9e9a3aaf5a486438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.572798 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 1056fad606894561bd417d1586cf8032 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.572775 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb726fc00544504a34a37bf3226f889 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.572993 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1056fad606894561bd417d1586cf8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.572999 25021 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:31.573043 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb726fc00544504a34a37bf3226f889. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.573849 25020 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:31.573879 25021 tablet_bootstrap.cc:654] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.574662 25020 tablet_bootstrap.cc:654] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.574846 25027 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:31.575291 25022 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:31.575307 25021 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:31.575326 24410 tablet_service.cc:1467] Processing CreateTablet for tablet 1056fad606894561bd417d1586cf8032 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.575381 25021 ts_tablet_manager.cc:1367] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.575546 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1056fad606894561bd417d1586cf8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.575829 25021 raft_consensus.cc:348] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.575919 25021 raft_consensus.cc:374] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.575946 25021 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.576059 25021 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.576097 25022 tablet_bootstrap.cc:654] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.576319 25021 ts_tablet_manager.cc:1398] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:31.577282 25027 tablet_bootstrap.cc:654] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.577530 25022 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:31.577612 25022 ts_tablet_manager.cc:1367] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.577965 25022 raft_consensus.cc:348] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.578053 25022 raft_consensus.cc:374] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.578086 25022 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.578235 25022 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.578564 25022 ts_tablet_manager.cc:1398] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.578676 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 1056fad606894561bd417d1586cf8032 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.578900 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1056fad606894561bd417d1586cf8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.579154 24148 tablet_service.cc:1467] Processing CreateTablet for tablet c82547c4a0a8475b9e9a3aaf5a486438 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf76b2f2a9e64ef6b4a0e27fcc1b66ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:31.579380 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82547c4a0a8475b9e9a3aaf5a486438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:31.580003 25022 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:31.580950 25022 tablet_bootstrap.cc:654] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.581482 25021 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:31.581503 25020 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:31.581589 25020 ts_tablet_manager.cc:1367] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 07:59:31.581991 25027 tablet_bootstrap.cc:492] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:31.582098 25027 ts_tablet_manager.cc:1367] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:31.582077 25020 raft_consensus.cc:348] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.582204 25020 raft_consensus.cc:374] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.582247 25020 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.582402 25021 tablet_bootstrap.cc:654] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.582379 25020 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.582613 25020 ts_tablet_manager.cc:1398] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.582577 25027 raft_consensus.cc:348] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.582674 25027 raft_consensus.cc:374] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.582724 25027 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.582851 25027 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.583240 25027 ts_tablet_manager.cc:1398] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.583398 25027 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:31.584204 25027 tablet_bootstrap.cc:654] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.584272 25021 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:31.584352 25021 ts_tablet_manager.cc:1367] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:31.584769 25021 raft_consensus.cc:348] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.584858 25021 raft_consensus.cc:374] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.584894 25021 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.585170 25022 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:31.585608 25021 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.585662 25027 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:31.585745 25027 ts_tablet_manager.cc:1367] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.585924 25021 ts_tablet_manager.cc:1398] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.586076 25021 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:31.586105 25022 ts_tablet_manager.cc:1367] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:31.586656 25022 raft_consensus.cc:348] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.586760 25022 raft_consensus.cc:374] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.586805 25022 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.586915 25022 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.587458 25027 raft_consensus.cc:348] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.587550 25027 raft_consensus.cc:374] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.587590 25027 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.587814 25022 ts_tablet_manager.cc:1398] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:31.587962 25027 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.588249 25041 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.588318 25041 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.588719 25041 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:31.589087 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:31.589159 25027 ts_tablet_manager.cc:1398] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:31.589196 24428 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.589462 24233 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.589618 25041 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.589655 25041 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.589689 25041 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.590507 25041 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.590853 25041 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:31.591231 25027 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:31.591276 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:31.591380 24428 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.591394 25021 tablet_bootstrap.cc:654] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.592159 25027 tablet_bootstrap.cc:654] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:31.592404 24428 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:31.592689 24233 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.592866 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:31.592933 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:31.592993 24168 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.593186 25041 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.593281 25041 raft_consensus.cc:686] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:31.593394 25041 consensus_queue.cc:237] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:31.593823 25027 tablet_bootstrap.cc:492] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:31.593910 25027 ts_tablet_manager.cc:1367] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:31.594308 25027 raft_consensus.cc:348] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.594421 25027 raft_consensus.cc:374] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.594458 25027 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.594502 25021 tablet_bootstrap.cc:492] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:31.594576 25021 ts_tablet_manager.cc:1367] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:31.594559 25027 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.594787 25027 ts_tablet_manager.cc:1398] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.595031 25021 raft_consensus.cc:348] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.594925 23891 catalog_manager.cc:5484] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:31.595126 25021 raft_consensus.cc:374] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:31.595167 25021 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:31.595296 25021 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.595575 25021 ts_tablet_manager.cc:1398] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:31.669775 25105 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.669903 25105 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.670329 25105 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.670748 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:31.670748 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:31.670922 24168 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:31.670922 24299 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:31.671247 23973 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.671429 25105 raft_consensus.cc:2781] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.671509 25105 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.671545 25105 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.672789 25105 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.673152 25105 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.673444 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:31.673537 24299 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.673503 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:31.673633 24168 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.674294 24299 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:31.674463 24168 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:31.674553 23973 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.674808 25105 raft_consensus.cc:2781] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.674878 25105 raft_consensus.cc:686] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:31.675032 25105 consensus_queue.cc:237] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.676734 23891 catalog_manager.cc:5484] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:31.807818 25041 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:31.807960 25041 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.808348 25041 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.808794 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:31.808800 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:31.808930 24428 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.808934 24168 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:31.809242 24232 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.809402 25041 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:31.809455 25041 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:31.809481 25041 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.810252 25041 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.810540 25041 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:31.810921 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:31.810920 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:31.811053 24428 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.811054 24168 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:31.811961 24168 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:31.811975 24428 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:31.812220 24232 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:31.812366 25041 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:31.812443 25041 raft_consensus.cc:686] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:31.812601 25041 consensus_queue.cc:237] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:31.814106 23891 catalog_manager.cc:5484] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:31.973791 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:31.974095 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:31.977118 25121 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:59:32.029513 24168 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.030753 25041 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.034816 25041 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.070827 24168 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.070829 24428 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.071314 25041 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.071518 25037 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.076788 24167 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.077189 25105 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.077255 24299 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.078016 25105 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.652 ms
time per row: 0.0588258 ms
I20240501 07:59:32.141183 23893 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:32.146777 24015 tablet_service.cc:1467] Processing CreateTablet for tablet ebf9d5a354ac4ef8a8df43802063eb4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.146867 24148 tablet_service.cc:1467] Processing CreateTablet for tablet ebf9d5a354ac4ef8a8df43802063eb4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.147167 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf9d5a354ac4ef8a8df43802063eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.147186 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf9d5a354ac4ef8a8df43802063eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.148115 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 090ee3c1fc884691810492f5eccbef67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.148416 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090ee3c1fc884691810492f5eccbef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.150488 25181 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:32.151226 24148 tablet_service.cc:1467] Processing CreateTablet for tablet b14500c7bb754282a0717ccd4257f6eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.151436 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14500c7bb754282a0717ccd4257f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.151477 25181 tablet_bootstrap.cc:654] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.151820 24410 tablet_service.cc:1467] Processing CreateTablet for tablet ebf9d5a354ac4ef8a8df43802063eb4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.152097 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf9d5a354ac4ef8a8df43802063eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.153820 25183 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:32.153959 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 090ee3c1fc884691810492f5eccbef67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.154152 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090ee3c1fc884691810492f5eccbef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.154599 25183 tablet_bootstrap.cc:654] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.155191 25179 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:32.155680 25183 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:32.155750 25183 ts_tablet_manager.cc:1367] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:32.155771 24408 tablet_service.cc:1467] Processing CreateTablet for tablet b14500c7bb754282a0717ccd4257f6eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.156002 25179 tablet_bootstrap.cc:654] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.156008 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14500c7bb754282a0717ccd4257f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.156229 25183 raft_consensus.cc:348] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.156332 25183 raft_consensus.cc:374] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.156366 25183 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.156512 25183 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.156839 25183 ts_tablet_manager.cc:1398] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:32.156958 25183 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:32.157702 25183 tablet_bootstrap.cc:654] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.158883 25183 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:32.158881 24017 tablet_service.cc:1467] Processing CreateTablet for tablet b14500c7bb754282a0717ccd4257f6eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.158964 25183 ts_tablet_manager.cc:1367] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:32.159168 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14500c7bb754282a0717ccd4257f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.158888 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 090ee3c1fc884691810492f5eccbef67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=8f2582088f384710b1d2f237adfb2d7e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.159402 25183 raft_consensus.cc:348] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.159505 25183 raft_consensus.cc:374] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.159539 25183 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.159629 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090ee3c1fc884691810492f5eccbef67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.159647 25183 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.159860 25183 ts_tablet_manager.cc:1398] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.160022 25183 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:32.160874 25183 tablet_bootstrap.cc:654] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.161543 25179 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:32.161605 25179 ts_tablet_manager.cc:1367] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:32.162027 25180 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:32.161998 25179 raft_consensus.cc:348] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.162091 25179 raft_consensus.cc:374] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.162142 25179 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.162247 25183 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:32.162325 25183 ts_tablet_manager.cc:1367] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.162299 25179 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.162650 25179 ts_tablet_manager.cc:1398] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.162794 25179 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:32.163300 25183 raft_consensus.cc:348] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.163412 25183 raft_consensus.cc:374] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.163442 25183 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.163570 25183 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.163667 25179 tablet_bootstrap.cc:654] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.164084 25180 tablet_bootstrap.cc:654] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.164682 25183 ts_tablet_manager.cc:1398] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.165969 25180 tablet_bootstrap.cc:492] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:32.165985 25181 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:32.166055 25180 ts_tablet_manager.cc:1367] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:32.166061 25181 ts_tablet_manager.cc:1367] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:32.166472 25181 raft_consensus.cc:348] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.166561 25181 raft_consensus.cc:374] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.166589 25181 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.166515 25180 raft_consensus.cc:348] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.166630 25180 raft_consensus.cc:374] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.166661 25180 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.166698 25181 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.166817 25180 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.166941 25181 ts_tablet_manager.cc:1398] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.167176 25180 ts_tablet_manager.cc:1398] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.167328 25180 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:32.168159 25180 tablet_bootstrap.cc:654] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.169207 25180 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:32.169317 25180 ts_tablet_manager.cc:1367] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.169865 25180 raft_consensus.cc:348] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.169962 25180 raft_consensus.cc:374] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.169996 25180 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.170140 25180 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.170497 25180 ts_tablet_manager.cc:1398] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.171185 25179 tablet_bootstrap.cc:492] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:32.171264 25179 ts_tablet_manager.cc:1367] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:32.171644 25179 raft_consensus.cc:348] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.171741 25179 raft_consensus.cc:374] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.171775 25179 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.171878 25179 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.172128 25179 ts_tablet_manager.cc:1398] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.172269 25179 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:32.173097 25179 tablet_bootstrap.cc:654] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.174060 25179 tablet_bootstrap.cc:492] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:32.174141 25179 ts_tablet_manager.cc:1367] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.174499 25179 raft_consensus.cc:348] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.174579 25179 raft_consensus.cc:374] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.174611 25179 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.174726 25179 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.174926 25179 ts_tablet_manager.cc:1398] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.200543 25029 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.200666 25029 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.201122 25029 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:32.201467 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:32.201545 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:32.201608 24037 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:32.201696 24428 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:32.201905 24101 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:32.202059 25029 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.202119 25029 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.202163 25029 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.202910 25029 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.203233 25029 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:32.203542 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:32.203570 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:32.203634 24037 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.203681 24428 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.204332 24037 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:32.204407 24428 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:32.204586 24101 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:32.204775 25029 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.204949 25029 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:32.205082 25029 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.206526 23895 catalog_manager.cc:5484] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:32.279027 25105 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.279156 25105 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.279654 25105 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:32.280129 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:32.280303 24167 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:32.280644 23970 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:32.280834 25105 raft_consensus.cc:2781] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.280891 25105 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.280921 25105 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.281966 25105 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.282327 25105 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:32.282658 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:32.282758 24167 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.283574 24167 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:32.283828 23970 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:32.283979 25105 raft_consensus.cc:2781] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.284056 25105 raft_consensus.cc:686] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:32.284201 25105 consensus_queue.cc:237] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.284430 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:32.284559 24430 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.284957 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:32.285558 24430 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:32.286043 23895 catalog_manager.cc:5484] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:32.304212 25147 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.304337 25147 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.305317 25147 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:32.308724 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:32.308874 24037 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:32.308857 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:32.308972 24299 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:32.309226 24363 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:32.309600 25147 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.309655 25147 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.309681 25147 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.310463 25147 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.310796 25147 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:32.311118 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:32.311168 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:32.311226 24037 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.311311 24299 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.311928 24037 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:32.312089 24299 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:32.312212 24363 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:32.312381 25147 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.312523 25147 raft_consensus.cc:686] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:32.312805 25147 consensus_queue.cc:237] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:32.314760 23895 catalog_manager.cc:5484] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:32.448587 25200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:32.448838 25200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:32.451838 25200 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:59:32.545012 24167 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.546185 24430 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.546718 25026 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.546991 25026 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.550324 24428 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.550794 24037 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.551193 25192 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.552809 24299 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.553258 25147 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.553602 25029 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.559216 24036 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.559860 25147 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.094 ms
time per row: 0.0590472 ms
I20240501 07:59:32.611954 23892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:32.617574 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 92db375d17784d2197a1f0f427e2d430 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.617863 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 92db375d17784d2197a1f0f427e2d430 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.617964 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db375d17784d2197a1f0f427e2d430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.618227 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db375d17784d2197a1f0f427e2d430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.618276 24147 tablet_service.cc:1467] Processing CreateTablet for tablet b21809c44bb940ffb007041977ac5696 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.618492 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21809c44bb940ffb007041977ac5696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.619939 25181 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:32.620208 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 92db375d17784d2197a1f0f427e2d430 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.620462 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 609f9ef5e1fa42daabc3c8fb9fb00613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.620641 25180 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:32.620970 25181 tablet_bootstrap.cc:654] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.621660 25180 tablet_bootstrap.cc:654] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.621824 24278 tablet_service.cc:1467] Processing CreateTablet for tablet b21809c44bb940ffb007041977ac5696 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.622076 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21809c44bb940ffb007041977ac5696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.622264 24409 tablet_service.cc:1467] Processing CreateTablet for tablet b21809c44bb940ffb007041977ac5696 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.622686 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21809c44bb940ffb007041977ac5696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.623376 25180 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:32.623469 25180 ts_tablet_manager.cc:1367] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.623979 25180 raft_consensus.cc:348] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.624097 25180 raft_consensus.cc:374] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.624132 25180 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.624269 25180 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.624600 25180 ts_tablet_manager.cc:1398] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.624786 25180 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:32.625109 24408 tablet_service.cc:1467] Processing CreateTablet for tablet 609f9ef5e1fa42daabc3c8fb9fb00613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.625346 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609f9ef5e1fa42daabc3c8fb9fb00613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.626358 25180 tablet_bootstrap.cc:654] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.626606 25183 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:32.627413 25183 tablet_bootstrap.cc:654] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.620486 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db375d17784d2197a1f0f427e2d430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.627743 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609f9ef5e1fa42daabc3c8fb9fb00613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.627957 25180 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:32.628038 25180 ts_tablet_manager.cc:1367] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:32.628371 25183 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:32.628438 25183 ts_tablet_manager.cc:1367] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.628480 25180 raft_consensus.cc:348] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.628585 25180 raft_consensus.cc:374] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.628625 25180 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.628752 25180 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.628847 25183 raft_consensus.cc:348] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.628939 25183 raft_consensus.cc:374] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.628973 25183 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.629093 25180 ts_tablet_manager.cc:1398] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.629110 25183 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.629360 25183 ts_tablet_manager.cc:1398] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.629963 25179 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:32.630928 25183 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:32.630945 25179 tablet_bootstrap.cc:654] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.631953 25183 tablet_bootstrap.cc:654] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.632558 25179 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:32.632649 25179 ts_tablet_manager.cc:1367] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:32.632758 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 609f9ef5e1fa42daabc3c8fb9fb00613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=675c80a0b4cd489194c71c7ba2f660fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:32.632958 25183 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:32.633011 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609f9ef5e1fa42daabc3c8fb9fb00613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:32.633025 25183 ts_tablet_manager.cc:1367] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.633172 25179 raft_consensus.cc:348] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.633296 25179 raft_consensus.cc:374] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.633332 25179 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.633448 25179 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.633448 25183 raft_consensus.cc:348] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.633530 25183 raft_consensus.cc:374] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.633556 25183 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.633656 25183 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.633750 25179 ts_tablet_manager.cc:1398] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.633913 25179 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:32.633946 25183 ts_tablet_manager.cc:1398] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.634904 25179 tablet_bootstrap.cc:654] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.635972 25179 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:32.636052 25179 ts_tablet_manager.cc:1367] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.636600 25179 raft_consensus.cc:348] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.636709 25179 raft_consensus.cc:374] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.636741 25179 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.636863 25181 tablet_bootstrap.cc:492] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:32.636858 25179 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.636983 25181 ts_tablet_manager.cc:1367] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240501 07:59:32.637053 25179 ts_tablet_manager.cc:1398] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.637449 25181 raft_consensus.cc:348] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.637568 25181 raft_consensus.cc:374] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.637619 25181 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.637746 25181 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.638054 25181 ts_tablet_manager.cc:1398] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:32.638254 25181 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:32.639364 25181 tablet_bootstrap.cc:654] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.640501 25181 tablet_bootstrap.cc:492] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:32.640575 25181 ts_tablet_manager.cc:1367] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:32.641172 25181 raft_consensus.cc:348] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.641281 25181 raft_consensus.cc:374] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.641335 25181 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.641494 25181 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.641758 25181 ts_tablet_manager.cc:1398] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:32.641997 25181 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:32.642849 25181 tablet_bootstrap.cc:654] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:32.643848 25181 tablet_bootstrap.cc:492] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:32.643916 25181 ts_tablet_manager.cc:1367] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:32.644342 25181 raft_consensus.cc:348] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.644436 25181 raft_consensus.cc:374] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:32.644475 25181 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:32.644608 25181 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.644856 25181 ts_tablet_manager.cc:1398] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:32.660032 25164 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.660137 25164 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.660570 25164 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:32.660921 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:32.661003 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:32.661074 24037 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:32.661157 24168 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:32.661387 24232 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:32.661574 25164 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.661638 25164 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.661696 25164 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.662580 25164 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.662935 25164 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:32.663276 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:32.663303 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:32.663401 24037 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.663419 24168 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.664206 24037 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:32.664224 24168 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:32.664451 24232 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:32.664628 25164 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.664703 25164 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:32.664839 25164 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:32.666250 23894 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:32.737285 25032 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.737407 25032 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.737859 25032 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:32.738276 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:32.738281 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:32.738415 24037 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:32.738426 24299 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:32.738714 24363 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:32.738890 25032 raft_consensus.cc:2781] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.738945 25032 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.738972 25032 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.739809 25032 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.740094 25032 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:32.740386 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:32.740399 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:32.740491 24299 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.740499 24037 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.741181 24299 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:32.741256 24037 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:32.741485 24366 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:32.741645 25032 raft_consensus.cc:2781] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.741751 25032 raft_consensus.cc:686] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:32.741888 25032 consensus_queue.cc:237] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:32.743587 23894 catalog_manager.cc:5484] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:32.761341 25164 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:32.761471 25164 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.761893 25164 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:32.762264 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:32.762274 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:32.762403 24168 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:32.762418 24428 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:32.762691 24232 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:32.762866 25164 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:32.762929 25164 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:32.762972 25164 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.763815 25164 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.764138 25164 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:32.764506 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:32.764495 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:32.764605 24168 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.764621 24428 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:32.765326 24168 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:32.765372 24428 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:32.765656 24233 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:32.765808 25164 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:32.765915 25164 raft_consensus.cc:686] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:32.766052 25164 consensus_queue.cc:237] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:32.767603 23894 catalog_manager.cc:5484] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
W20240501 07:59:32.889132 25251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:32.889393 25251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:32.892467 25251 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:59:32.973738 24168 raft_consensus.cc:1261] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.974212 25164 consensus_queue.cc:1035] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.975814 24037 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.976025 24428 raft_consensus.cc:1261] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.976218 25032 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.980525 24299 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.980717 25037 consensus_queue.cc:1035] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.980871 25032 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.989691 24168 raft_consensus.cc:1261] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.991747 24036 raft_consensus.cc:1261] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:32.991819 25037 consensus_queue.cc:1035] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:32.997607 25283 consensus_queue.cc:1035] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.093 ms
time per row: 0.0510467 ms
I20240501 07:59:33.042409 23889 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:33.047823 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4f9ceedafe4c90ac182f1ba61d583a (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.047963 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4f9ceedafe4c90ac182f1ba61d583a (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.048115 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4f9ceedafe4c90ac182f1ba61d583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.048290 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4f9ceedafe4c90ac182f1ba61d583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.048676 24278 tablet_service.cc:1467] Processing CreateTablet for tablet 15ff4003c1654707a17d65993fb938af (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.048930 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ff4003c1654707a17d65993fb938af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.049785 25180 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:33.049980 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4f9ceedafe4c90ac182f1ba61d583a (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.050194 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4f9ceedafe4c90ac182f1ba61d583a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.050637 25180 tablet_bootstrap.cc:654] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.050856 25181 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:33.051226 25179 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:33.051745 25181 tablet_bootstrap.cc:654] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.051702 24408 tablet_service.cc:1467] Processing CreateTablet for tablet c6313ef646004e3eba64205bd4f535ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.051960 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6313ef646004e3eba64205bd4f535ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.052081 25179 tablet_bootstrap.cc:654] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.052913 25180 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:33.052996 25180 ts_tablet_manager.cc:1367] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:33.053216 25179 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:33.053285 25179 ts_tablet_manager.cc:1367] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:33.053452 25180 raft_consensus.cc:348] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.053545 25180 raft_consensus.cc:374] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.053575 25180 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.053683 25179 raft_consensus.cc:348] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.053715 25180 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.053768 25179 raft_consensus.cc:374] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.053773 25183 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:33.053799 25179 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.053906 25179 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.053933 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 15ff4003c1654707a17d65993fb938af (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.053989 25180 ts_tablet_manager.cc:1398] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:33.054144 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ff4003c1654707a17d65993fb938af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.054189 25179 ts_tablet_manager.cc:1398] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.054600 25183 tablet_bootstrap.cc:654] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.057307 24015 tablet_service.cc:1467] Processing CreateTablet for tablet c6313ef646004e3eba64205bd4f535ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.057364 24148 tablet_service.cc:1467] Processing CreateTablet for tablet c6313ef646004e3eba64205bd4f535ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.057555 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6313ef646004e3eba64205bd4f535ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.057586 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6313ef646004e3eba64205bd4f535ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.058647 25181 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:33.058684 25183 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:33.058738 25181 ts_tablet_manager.cc:1367] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:33.058766 25183 ts_tablet_manager.cc:1367] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:59:33.059187 25181 raft_consensus.cc:348] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.059317 25181 raft_consensus.cc:374] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.059352 25181 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.059463 25181 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.059510 25183 raft_consensus.cc:348] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.059587 25179 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:33.059614 25183 raft_consensus.cc:374] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.059652 25183 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.059727 25181 ts_tablet_manager.cc:1398] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.059772 25183 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.059875 25181 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:33.059973 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 15ff4003c1654707a17d65993fb938af (DEFAULT_TABLE table=rebalance_test_table_4 [id=71c669c608324d46a3cdc3d785e8de3b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.060074 25183 ts_tablet_manager.cc:1398] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.060207 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ff4003c1654707a17d65993fb938af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.060245 25183 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:33.060420 25179 tablet_bootstrap.cc:654] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.060778 25181 tablet_bootstrap.cc:654] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.061216 25183 tablet_bootstrap.cc:654] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.061743 25181 tablet_bootstrap.cc:492] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:33.061806 25181 ts_tablet_manager.cc:1367] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:33.062110 25181 raft_consensus.cc:348] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.062186 25181 raft_consensus.cc:374] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.062216 25181 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.062343 25181 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.062626 25181 ts_tablet_manager.cc:1398] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.063555 25180 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:33.064733 25180 tablet_bootstrap.cc:654] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.066239 25180 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:33.066347 25180 ts_tablet_manager.cc:1367] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:33.066866 25180 raft_consensus.cc:348] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.067034 25180 raft_consensus.cc:374] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.067088 25180 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.067230 25180 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.067503 25180 ts_tablet_manager.cc:1398] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:33.068161 25179 tablet_bootstrap.cc:492] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:33.068254 25179 ts_tablet_manager.cc:1367] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:33.068310 25183 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:33.068393 25183 ts_tablet_manager.cc:1367] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:33.068795 25179 raft_consensus.cc:348] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.068912 25179 raft_consensus.cc:374] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.068876 25183 raft_consensus.cc:348] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.068953 25179 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.068996 25183 raft_consensus.cc:374] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.069061 25183 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.069092 25179 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.069389 25179 ts_tablet_manager.cc:1398] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:33.069330 25183 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.069561 25179 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:33.069691 25183 ts_tablet_manager.cc:1398] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.070423 25179 tablet_bootstrap.cc:654] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.071424 25179 tablet_bootstrap.cc:492] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:33.071501 25179 ts_tablet_manager.cc:1367] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:33.072136 25179 raft_consensus.cc:348] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.072263 25179 raft_consensus.cc:374] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.072301 25179 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.072661 25179 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.072978 25179 ts_tablet_manager.cc:1398] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.134887 25032 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.135017 25032 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.135386 25032 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:33.135759 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:33.135746 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:33.135897 24036 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.135915 24299 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.136193 24363 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:33.136389 25032 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.136456 25032 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.136505 25032 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.137465 25032 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.137784 25032 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:33.138100 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:33.138209 24299 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.138938 24299 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:33.139140 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:33.139231 24366 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.139271 24036 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.139418 25032 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.139487 25032 raft_consensus.cc:686] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:33.139638 25032 consensus_queue.cc:237] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.140180 24036 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:33.141605 23894 catalog_manager.cc:5484] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:33.231170 25032 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.231312 25032 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.231787 25032 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:33.232173 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:33.232322 24036 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.232664 24363 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:33.232836 25032 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.232887 25032 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.232933 25032 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.233721 25032 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.234055 25032 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:33.234426 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:33.234560 24036 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.235469 24036 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:33.235742 24363 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:33.235720 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:33.235762 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:33.235844 24168 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.235894 25032 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.235987 25032 raft_consensus.cc:686] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:33.236145 25032 consensus_queue.cc:237] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.236238 25283 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.236351 25283 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.236709 25283 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:33.236970 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:33.237087 24168 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:33.237123 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:33.237339 24232 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.237412 24036 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:33.237491 25283 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.237535 25283 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.237569 25283 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.237990 23894 catalog_manager.cc:5484] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:33.238531 25283 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.238808 25283 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:33.239141 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:33.239158 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:33.239251 24036 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.239251 24168 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.240077 24036 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:33.240084 24168 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:33.240373 24232 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.240512 25283 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.240586 25283 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:33.240702 25283 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.242257 23894 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
W20240501 07:59:33.352326 25302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:33.352592 25302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:33.355702 25302 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:59:33.446190 24036 raft_consensus.cc:1261] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.446296 24168 raft_consensus.cc:1261] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.446782 25164 consensus_queue.cc:1035] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.447986 24299 raft_consensus.cc:1261] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.448783 25147 consensus_queue.cc:1035] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.449892 25283 consensus_queue.cc:1035] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.453392 24167 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.454618 24167 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.459712 24037 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.459779 25032 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.460331 24036 raft_consensus.cc:1261] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:33.460577 25032 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.460904 25032 consensus_queue.cc:1035] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.477504 25235 mvcc.cc:205] Tried to move back new op lower bound from 7022798329661870080 to 7022798328381325312. Current Snapshot: MvccSnapshot[applied={T|T < 7022798329661870080}]
INSERT report
rows total: 2000
time total: 115.368 ms
time per row: 0.0576838 ms
I20240501 07:59:33.514771 23888 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:33.520519 24148 tablet_service.cc:1467] Processing CreateTablet for tablet e786d8c542154add8bf34774a6ef095d (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.520530 24277 tablet_service.cc:1467] Processing CreateTablet for tablet e786d8c542154add8bf34774a6ef095d (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.520828 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d8c542154add8bf34774a6ef095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.520828 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d8c542154add8bf34774a6ef095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.522927 25181 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:33.522994 25180 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:33.523309 24408 tablet_service.cc:1467] Processing CreateTablet for tablet e1f318e2ba0c4a26ad4cd8630bb0748c (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.523572 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f318e2ba0c4a26ad4cd8630bb0748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.523872 25181 tablet_bootstrap.cc:654] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.523916 25180 tablet_bootstrap.cc:654] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.524374 24409 tablet_service.cc:1467] Processing CreateTablet for tablet e786d8c542154add8bf34774a6ef095d (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.524605 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e786d8c542154add8bf34774a6ef095d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.525328 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 80205bf3beff4e3bba804332a6aa8c73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.525610 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80205bf3beff4e3bba804332a6aa8c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.526512 25183 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:33.527328 25183 tablet_bootstrap.cc:654] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.528438 25179 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:33.529103 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 80205bf3beff4e3bba804332a6aa8c73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.529220 25179 tablet_bootstrap.cc:654] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.529281 25180 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:33.529332 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80205bf3beff4e3bba804332a6aa8c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.529359 25180 ts_tablet_manager.cc:1367] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:59:33.529835 25180 raft_consensus.cc:348] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.529944 25180 raft_consensus.cc:374] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.529978 25180 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.530104 25180 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.530367 25180 ts_tablet_manager.cc:1398] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:33.530431 24278 tablet_service.cc:1467] Processing CreateTablet for tablet 80205bf3beff4e3bba804332a6aa8c73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.530652 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80205bf3beff4e3bba804332a6aa8c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.531499 25180 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:33.532246 25180 tablet_bootstrap.cc:654] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.532513 24148 tablet_service.cc:1467] Processing CreateTablet for tablet e1f318e2ba0c4a26ad4cd8630bb0748c (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.532712 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f318e2ba0c4a26ad4cd8630bb0748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.533483 24277 tablet_service.cc:1467] Processing CreateTablet for tablet e1f318e2ba0c4a26ad4cd8630bb0748c (DEFAULT_TABLE table=rebalance_test_table_5 [id=90082aafa52f42baac155ad0cf3d1834]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:33.533720 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f318e2ba0c4a26ad4cd8630bb0748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:33.535773 25183 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:33.535851 25183 ts_tablet_manager.cc:1367] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:33.536259 25183 raft_consensus.cc:348] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.536358 25183 raft_consensus.cc:374] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.536386 25183 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.536491 25183 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.536739 25183 ts_tablet_manager.cc:1398] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:33.537364 25179 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:33.537371 25180 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:33.537443 25180 ts_tablet_manager.cc:1367] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:33.537454 25179 ts_tablet_manager.cc:1367] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:33.537469 25183 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:33.537835 25179 raft_consensus.cc:348] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.537912 25179 raft_consensus.cc:374] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.537932 25179 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.537878 25180 raft_consensus.cc:348] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.537981 25180 raft_consensus.cc:374] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.538008 25180 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.538035 25179 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.538125 25180 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.538300 25179 ts_tablet_manager.cc:1398] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.538365 25183 tablet_bootstrap.cc:654] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.538411 25180 ts_tablet_manager.cc:1398] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:33.538573 25180 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:33.539443 25180 tablet_bootstrap.cc:654] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.540299 25181 tablet_bootstrap.cc:492] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:33.540386 25181 ts_tablet_manager.cc:1367] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:59:33.540833 25181 raft_consensus.cc:348] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.540940 25181 raft_consensus.cc:374] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.540980 25181 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.541107 25181 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.541404 25181 ts_tablet_manager.cc:1398] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.541571 25181 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:33.542377 25181 tablet_bootstrap.cc:654] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.542536 25180 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:33.542598 25180 ts_tablet_manager.cc:1367] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:33.542943 25180 raft_consensus.cc:348] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.543043 25180 raft_consensus.cc:374] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.543072 25180 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.543175 25180 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.543437 25180 ts_tablet_manager.cc:1398] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.544965 25183 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:33.545046 25183 ts_tablet_manager.cc:1367] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 07:59:33.545403 25181 tablet_bootstrap.cc:492] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:33.545467 25181 ts_tablet_manager.cc:1367] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:33.545475 25183 raft_consensus.cc:348] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.545580 25183 raft_consensus.cc:374] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.545624 25183 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.545737 25183 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.545830 25181 raft_consensus.cc:348] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.545925 25181 raft_consensus.cc:374] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.545955 25181 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.546025 25183 ts_tablet_manager.cc:1398] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:33.546062 25181 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.548173 25181 ts_tablet_manager.cc:1398] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:33.548386 25181 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:33.549407 25181 tablet_bootstrap.cc:654] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:33.550729 25032 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.550802 25032 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.551112 25032 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:33.551177 25181 tablet_bootstrap.cc:492] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:33.551251 25181 ts_tablet_manager.cc:1367] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:33.551533 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:33.551656 24168 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.551645 25181 raft_consensus.cc:348] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.551743 25181 raft_consensus.cc:374] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:33.551781 25181 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:33.551867 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:33.551929 25181 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.551972 24298 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:33.552285 25181 ts_tablet_manager.cc:1398] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:33.552280 24366 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.552461 25329 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.552511 25329 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.552541 25329 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.553457 25329 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.553797 25329 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:33.554067 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:33.554149 24298 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.554244 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:33.554360 24168 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.555095 24298 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:33.555320 24168 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:33.555368 24366 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.555511 25329 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.555591 25329 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:33.555711 25329 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:33.557253 23886 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:33.850966 25105 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.851140 25105 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.851538 25105 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:33.852028 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:33.852026 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:33.852198 24298 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:33.852195 24168 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:33.852505 23973 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:33.852677 25105 raft_consensus.cc:2781] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.852738 25105 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.852792 25105 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.853617 25105 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.853940 25105 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:33.854316 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:33.854324 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:33.854429 24298 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.854439 24168 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.855218 24298 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:33.855247 24168 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:33.855526 23970 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:33.855688 25105 raft_consensus.cc:2781] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.855794 25105 raft_consensus.cc:686] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:33.855926 25105 consensus_queue.cc:237] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:33.857733 23886 catalog_manager.cc:5484] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:33.872598 25335 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:33.872709 25335 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.873201 25335 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:33.873661 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:33.873814 24428 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:33.874137 24102 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:33.874337 25335 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:33.874408 25335 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:33.874439 25335 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.875496 25335 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.875854 25335 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:33.876181 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:33.876295 24428 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:33.876989 24428 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:33.877310 24102 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:33.877506 25335 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:33.877589 25335 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:33.877727 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:33.877720 25335 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:33.877734 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:33.877859 24298 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:33.879421 23886 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:33.963064 25329 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:33.966253 25329 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:34.010725 25350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:34.011047 25350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:34.014021 25350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 07:59:34.098562 24298 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.098881 24428 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.100206 24298 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.100854 25335 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.101153 25335 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.106278 24299 raft_consensus.cc:1261] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.106655 25105 consensus_queue.cc:1035] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.115626 24168 raft_consensus.cc:1261] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.116678 25105 consensus_queue.cc:1035] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.996 ms
time per row: 0.053498 ms
I20240501 07:59:34.165735 23886 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:34.171387 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9401efd19547c2ac46a58c72c25834 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.171730 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9401efd19547c2ac46a58c72c25834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.171624 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 130aef27115540ed8e0bc20158c1aeea (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.171927 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130aef27115540ed8e0bc20158c1aeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.172371 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9401efd19547c2ac46a58c72c25834 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.172628 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9401efd19547c2ac46a58c72c25834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.171625 24408 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9401efd19547c2ac46a58c72c25834 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.174530 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 130aef27115540ed8e0bc20158c1aeea (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.174614 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9401efd19547c2ac46a58c72c25834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.174775 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130aef27115540ed8e0bc20158c1aeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.174846 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 130aef27115540ed8e0bc20158c1aeea (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.175092 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130aef27115540ed8e0bc20158c1aeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.176425 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0272069f81436f81725c3092e0e382 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.176683 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0272069f81436f81725c3092e0e382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.176889 25409 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:34.177199 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0272069f81436f81725c3092e0e382 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.177457 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0272069f81436f81725c3092e0e382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.177966 25409 tablet_bootstrap.cc:654] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.178372 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0272069f81436f81725c3092e0e382 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d052338ba2494917acefaec8cfee36ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.178637 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0272069f81436f81725c3092e0e382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.179256 25407 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:34.180083 25408 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:34.180236 25407 tablet_bootstrap.cc:654] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.180301 25409 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:34.180392 25409 ts_tablet_manager.cc:1367] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.180912 25408 tablet_bootstrap.cc:654] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.180852 25409 raft_consensus.cc:348] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.180970 25409 raft_consensus.cc:374] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.181020 25409 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.181154 25409 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.181524 25409 ts_tablet_manager.cc:1398] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.181516 25406 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:34.181700 25407 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:34.181773 25407 ts_tablet_manager.cc:1367] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:34.182235 25407 raft_consensus.cc:348] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.182334 25407 raft_consensus.cc:374] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.182368 25407 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.182372 25406 tablet_bootstrap.cc:654] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.182523 25407 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.182843 25407 ts_tablet_manager.cc:1398] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.183122 25407 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:34.183954 25408 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:34.183986 25406 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:34.184072 25406 ts_tablet_manager.cc:1367] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:34.184147 25408 ts_tablet_manager.cc:1367] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.184343 25407 tablet_bootstrap.cc:654] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.184559 25406 raft_consensus.cc:348] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.184672 25406 raft_consensus.cc:374] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.184705 25406 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.184829 25406 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.185117 25406 ts_tablet_manager.cc:1398] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.185106 25408 raft_consensus.cc:348] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.185309 25408 raft_consensus.cc:374] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.185318 25406 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:34.185390 25408 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.185657 25408 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.186074 25408 ts_tablet_manager.cc:1398] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.186151 25406 tablet_bootstrap.cc:654] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.186364 25408 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:34.187275 25408 tablet_bootstrap.cc:654] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.187883 25407 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:34.187965 25407 ts_tablet_manager.cc:1367] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:34.188400 25407 raft_consensus.cc:348] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.188513 25407 raft_consensus.cc:374] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.188531 25408 tablet_bootstrap.cc:492] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:34.188561 25407 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.188607 25408 ts_tablet_manager.cc:1367] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:34.188699 25407 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.189035 25408 raft_consensus.cc:348] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.189121 25408 raft_consensus.cc:374] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.189153 25408 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.189272 25408 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.189471 25407 ts_tablet_manager.cc:1398] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.189496 25408 ts_tablet_manager.cc:1398] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.189666 25408 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:34.189683 25407 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:34.190325 25408 tablet_bootstrap.cc:654] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.190563 25407 tablet_bootstrap.cc:654] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.191428 25408 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:34.191526 25408 ts_tablet_manager.cc:1367] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.191994 25408 raft_consensus.cc:348] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.192097 25408 raft_consensus.cc:374] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.192127 25406 tablet_bootstrap.cc:492] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:34.192137 25408 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.192242 25406 ts_tablet_manager.cc:1367] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:34.192272 25408 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.192343 25407 tablet_bootstrap.cc:492] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:34.192492 25407 ts_tablet_manager.cc:1367] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:34.192560 25408 ts_tablet_manager.cc:1398] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.192731 25406 raft_consensus.cc:348] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.192842 25406 raft_consensus.cc:374] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.192883 25406 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.192998 25407 raft_consensus.cc:348] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.193001 25406 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.193094 25407 raft_consensus.cc:374] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.193217 25407 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.193296 25406 ts_tablet_manager.cc:1398] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.193347 25407 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.193615 25407 ts_tablet_manager.cc:1398] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.244419 25029 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.244539 25029 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.244865 25029 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.245280 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:34.245275 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:34.245404 24428 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:34.245419 24037 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:34.245678 24102 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:34.245836 25029 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.245895 25029 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.245930 25029 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.246793 25029 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.247129 25029 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.247493 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:34.247550 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:34.247617 24037 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.247656 24428 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.248610 24037 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:34.248610 24428 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:34.248870 24101 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:34.249037 25029 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.249127 25029 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:34.249365 25029 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.251281 23887 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:34.261634 25392 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.261757 25392 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.262123 25392 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.262535 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:34.262599 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:34.262692 24168 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:34.262712 24037 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:34.263120 24363 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:34.263306 25392 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.263368 25392 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.263419 25392 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.264215 25392 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.264523 25392 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.264813 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:34.264871 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:34.264918 24168 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.264972 24037 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.265651 24168 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:34.265662 24037 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:34.265903 24363 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:34.266109 25392 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.266209 25392 raft_consensus.cc:686] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:34.266351 25392 consensus_queue.cc:237] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.267989 23887 catalog_manager.cc:5484] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:34.303596 25283 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.303714 25283 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.304077 25283 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.304476 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:34.304476 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:34.304620 24428 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:34.304620 24037 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:34.304912 24233 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.305101 25283 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.305163 25283 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.305194 25283 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.306001 25283 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.306306 25283 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:34.306609 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:34.306608 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:34.306730 24428 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.306735 24037 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.307643 24037 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:34.307643 24428 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:34.307919 24233 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.308147 25283 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.308326 25283 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:34.308467 25283 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:34.310027 23887 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
W20240501 07:59:34.442677 25423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:34.442922 25423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:34.446154 25423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 07:59:34.532992 24037 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.533443 25283 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.540705 24428 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.541173 25421 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.546613 24036 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.546726 24430 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.547104 24167 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.547111 25156 mvcc.cc:205] Tried to move back new op lower bound from 7022798334083358720 to 7022798333168898048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798334083358720 or (T in {7022798334083358720})}]
I20240501 07:59:34.547498 24035 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.547977 25029 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.548645 25032 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.553361 25083 mvcc.cc:205] Tried to move back new op lower bound from 7022798334083358720 to 7022798333168898048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798334083358720}]
I20240501 07:59:34.554550 25029 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.556129 25329 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.573596 25162 mvcc.cc:205] Tried to move back new op lower bound from 7022798334136762368 to 7022798332996128768. Current Snapshot: MvccSnapshot[applied={T|T < 7022798334086156288}]
INSERT report
rows total: 2000
time total: 107.013 ms
time per row: 0.0535066 ms
I20240501 07:59:34.599603 23887 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:34.606204 24148 tablet_service.cc:1467] Processing CreateTablet for tablet af824adcc5ae43e79679d0257a4e578f (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.606256 24277 tablet_service.cc:1467] Processing CreateTablet for tablet af824adcc5ae43e79679d0257a4e578f (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.606511 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af824adcc5ae43e79679d0257a4e578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.606547 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af824adcc5ae43e79679d0257a4e578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.606828 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 02c7ed9d33204e06a2579bde5ca71960 (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.607057 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c7ed9d33204e06a2579bde5ca71960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.608373 25409 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:34.608655 25406 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:34.609220 25409 tablet_bootstrap.cc:654] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.609282 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 3119928e27b14eb0a68ae0aa0b70059c (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.609495 25406 tablet_bootstrap.cc:654] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.609560 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3119928e27b14eb0a68ae0aa0b70059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.609756 24015 tablet_service.cc:1467] Processing CreateTablet for tablet af824adcc5ae43e79679d0257a4e578f (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.609951 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af824adcc5ae43e79679d0257a4e578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.610230 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 02c7ed9d33204e06a2579bde5ca71960 (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.610432 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c7ed9d33204e06a2579bde5ca71960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.611253 24408 tablet_service.cc:1467] Processing CreateTablet for tablet 3119928e27b14eb0a68ae0aa0b70059c (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.611429 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3119928e27b14eb0a68ae0aa0b70059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.611603 25408 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:34.612159 24278 tablet_service.cc:1467] Processing CreateTablet for tablet 3119928e27b14eb0a68ae0aa0b70059c (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.612386 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3119928e27b14eb0a68ae0aa0b70059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.612463 25408 tablet_bootstrap.cc:654] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.612886 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 02c7ed9d33204e06a2579bde5ca71960 (DEFAULT_TABLE table=rebalance_test_table_7 [id=28ac19bf29c64bb9a1d08fc1839a0e87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:34.613095 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c7ed9d33204e06a2579bde5ca71960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:34.613513 25408 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:34.613582 25408 ts_tablet_manager.cc:1367] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.614097 25408 raft_consensus.cc:348] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.614203 25408 raft_consensus.cc:374] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.614231 25408 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.614351 25407 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:34.614337 25408 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.614585 25408 ts_tablet_manager.cc:1398] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.614745 25408 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:34.615207 25407 tablet_bootstrap.cc:654] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.615598 25408 tablet_bootstrap.cc:654] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.616605 25408 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:34.616698 25408 ts_tablet_manager.cc:1367] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:34.616961 25409 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:34.617048 25409 ts_tablet_manager.cc:1367] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:34.617149 25408 raft_consensus.cc:348] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.617249 25408 raft_consensus.cc:374] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.617283 25408 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.617414 25408 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.617522 25409 raft_consensus.cc:348] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.617619 25409 raft_consensus.cc:374] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.617656 25409 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.617712 25408 ts_tablet_manager.cc:1398] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.617776 25409 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.618053 25409 ts_tablet_manager.cc:1398] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.618238 25409 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:34.619098 25409 tablet_bootstrap.cc:654] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.619786 25407 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:34.619872 25407 ts_tablet_manager.cc:1367] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:34.619997 25406 tablet_bootstrap.cc:492] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:34.620054 25409 tablet_bootstrap.cc:492] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:34.620085 25406 ts_tablet_manager.cc:1367] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:34.620111 25409 ts_tablet_manager.cc:1367] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:34.620330 25407 raft_consensus.cc:348] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620445 25407 raft_consensus.cc:374] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.620482 25407 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.620496 25409 raft_consensus.cc:348] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620523 25406 raft_consensus.cc:348] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620605 25409 raft_consensus.cc:374] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.620638 25406 raft_consensus.cc:374] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.620635 25409 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.620620 25407 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620671 25406 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.620774 25409 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620802 25406 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.620904 25407 ts_tablet_manager.cc:1398] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.621037 25409 ts_tablet_manager.cc:1398] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.621069 25406 ts_tablet_manager.cc:1398] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:34.621191 25409 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:34.621208 25406 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:34.622046 25409 tablet_bootstrap.cc:654] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.622046 25406 tablet_bootstrap.cc:654] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.622051 25407 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:34.623212 25406 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:34.623368 25409 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:34.623445 25409 ts_tablet_manager.cc:1367] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:34.623641 25407 tablet_bootstrap.cc:654] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:34.623860 25409 raft_consensus.cc:348] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.623956 25409 raft_consensus.cc:374] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.623996 25409 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.624109 25409 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.624402 25409 ts_tablet_manager.cc:1398] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.625113 25406 ts_tablet_manager.cc:1367] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:34.625555 25406 raft_consensus.cc:348] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.625666 25406 raft_consensus.cc:374] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.625705 25406 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.625813 25406 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.626108 25406 ts_tablet_manager.cc:1398] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:34.627453 25407 tablet_bootstrap.cc:492] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:34.627538 25407 ts_tablet_manager.cc:1367] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 07:59:34.628037 25407 raft_consensus.cc:348] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.628139 25407 raft_consensus.cc:374] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:34.628176 25407 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:34.628288 25407 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.628589 25407 ts_tablet_manager.cc:1398] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:34.686607 25392 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.686714 25392 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.687201 25392 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:34.687485 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:34.687485 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:34.687623 24299 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:34.687623 24168 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:34.687904 24366 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.688158 25392 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.688228 25392 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.688268 25392 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.689095 25392 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.689409 25392 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:34.689726 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:34.689749 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:34.689851 24168 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.689875 24299 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.690785 24168 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:34.690835 24299 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:34.691166 24366 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.691314 25392 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.691421 25392 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:34.691567 25392 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:34.693441 23887 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:34.732247 25105 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.732386 25105 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.732811 25105 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:34.733294 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:34.733302 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:34.733422 24299 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:34.733460 24168 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:34.733712 23973 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.733897 25105 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.733963 25105 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.734014 25105 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.734793 25105 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.735150 25105 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:34.735484 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:34.735495 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:34.735589 24299 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.735600 24168 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.736274 24299 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:34.736375 24168 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:34.736549 23973 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:34.736697 25105 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.736774 25105 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:34.736908 25105 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.738775 23889 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:34.755664 25105 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.755790 25105 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.756325 25105 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:34.756664 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:34.756713 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:34.756865 24428 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:34.756903 24299 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 0.
I20240501 07:59:34.757155 23971 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:34.757339 25105 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:34.757412 25105 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:34.757463 25105 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.758199 25105 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.758488 25105 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:34.758829 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:34.758869 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:34.758944 24299 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.759023 24428 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:34.759725 24428 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:34.759984 23971 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:34.760140 25105 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:34.760278 25105 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:34.760310 24299 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 1.
I20240501 07:59:34.760620 25105 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:34.762221 23889 catalog_manager.cc:5484] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:34.874514 25474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:34.874742 25474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:34.877779 25474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 07:59:34.957753 24428 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.958084 24299 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.958393 24298 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.958551 25105 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.958863 25105 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.958932 25499 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.959620 24168 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.963266 25500 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.965658 24167 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.966248 25392 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:34.966580 24299 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:34.967195 25392 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.642 ms
time per row: 0.061321 ms
I20240501 07:59:35.042008 23891 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:35.047206 24147 tablet_service.cc:1467] Processing CreateTablet for tablet 6af81850415b45bd9f8c1d52044bd372 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.047533 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af81850415b45bd9f8c1d52044bd372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.047567 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5d8581db864dd89b7e292cb93670c1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.047837 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5d8581db864dd89b7e292cb93670c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.047907 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 6af81850415b45bd9f8c1d52044bd372 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.048154 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af81850415b45bd9f8c1d52044bd372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.049337 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5d8581db864dd89b7e292cb93670c1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.049394 24146 tablet_service.cc:1467] Processing CreateTablet for tablet 4da90005c90d4944996a8a2d2bf59c18 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.049611 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5d8581db864dd89b7e292cb93670c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.049908 24408 tablet_service.cc:1467] Processing CreateTablet for tablet 4da90005c90d4944996a8a2d2bf59c18 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.050005 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5d8581db864dd89b7e292cb93670c1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.050112 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90005c90d4944996a8a2d2bf59c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.050285 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5d8581db864dd89b7e292cb93670c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.051326 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 6af81850415b45bd9f8c1d52044bd372 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.051553 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af81850415b45bd9f8c1d52044bd372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.051746 25407 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:35.052012 25409 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:35.052429 25408 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:35.052816 25409 tablet_bootstrap.cc:654] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.052815 25407 tablet_bootstrap.cc:654] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.053500 25408 tablet_bootstrap.cc:654] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.053692 25406 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:35.053912 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 4da90005c90d4944996a8a2d2bf59c18 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3427e58597834102b1828e084b24d3c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.054129 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90005c90d4944996a8a2d2bf59c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.054402 24146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90005c90d4944996a8a2d2bf59c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.054519 25406 tablet_bootstrap.cc:654] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.055696 25407 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:35.055783 25407 ts_tablet_manager.cc:1367] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:35.056381 25407 raft_consensus.cc:348] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.056488 25407 raft_consensus.cc:374] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.056521 25407 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.056641 25407 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.056790 25409 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:35.056860 25409 ts_tablet_manager.cc:1367] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 07:59:35.056931 25407 ts_tablet_manager.cc:1398] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:35.057089 25407 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:35.057168 25408 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:35.057252 25408 ts_tablet_manager.cc:1367] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:35.057284 25409 raft_consensus.cc:348] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.057375 25409 raft_consensus.cc:374] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.057411 25409 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.057541 25409 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.057605 25408 raft_consensus.cc:348] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.057713 25408 raft_consensus.cc:374] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.057749 25408 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.057786 25409 ts_tablet_manager.cc:1398] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.057854 25408 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.057914 25407 tablet_bootstrap.cc:654] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.057955 25409 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:35.058112 25408 ts_tablet_manager.cc:1398] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.058813 25409 tablet_bootstrap.cc:654] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.058907 25407 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:35.058975 25407 ts_tablet_manager.cc:1367] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:35.059376 25408 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:35.059408 25407 raft_consensus.cc:348] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.059494 25407 raft_consensus.cc:374] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.059522 25407 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.059669 25407 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.060034 25407 ts_tablet_manager.cc:1398] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.060240 25408 tablet_bootstrap.cc:654] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.060808 25406 tablet_bootstrap.cc:492] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:35.060889 25406 ts_tablet_manager.cc:1367] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:35.061292 25406 raft_consensus.cc:348] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.061399 25406 raft_consensus.cc:374] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.061431 25406 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.061544 25406 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.061796 25406 ts_tablet_manager.cc:1398] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.061964 25406 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:35.062544 25408 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:35.062634 25408 ts_tablet_manager.cc:1367] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:35.063123 25406 tablet_bootstrap.cc:654] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.063092 25408 raft_consensus.cc:348] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.063295 25408 raft_consensus.cc:374] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.063341 25408 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.063457 25408 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.063743 25408 ts_tablet_manager.cc:1398] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.064294 25409 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:35.064381 25409 ts_tablet_manager.cc:1367] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:35.064967 25409 raft_consensus.cc:348] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.065070 25409 raft_consensus.cc:374] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.065099 25409 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.065217 25409 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.065490 25409 ts_tablet_manager.cc:1398] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.065543 25406 tablet_bootstrap.cc:492] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:35.065618 25406 ts_tablet_manager.cc:1367] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:35.066001 25406 raft_consensus.cc:348] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.066040 25029 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.066085 25406 raft_consensus.cc:374] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.066118 25029 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.066509 25029 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.066870 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:35.066121 25406 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.066982 24037 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:35.067049 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:35.067104 25406 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.067149 24430 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 0.
I20240501 07:59:35.067332 24101 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:35.067461 25029 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.067514 25029 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.067566 25029 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.068542 25029 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.068886 25029 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.069449 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:35.069572 24430 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.070281 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:35.070389 24037 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.071475 24037 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:35.071835 24101 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:35.071938 24430 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 1.
I20240501 07:59:35.072305 25406 ts_tablet_manager.cc:1398] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 07:59:35.072501 25406 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:35.072588 25029 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.072669 25029 raft_consensus.cc:686] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:35.072789 25029 consensus_queue.cc:237] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.073510 25406 tablet_bootstrap.cc:654] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.074625 25406 tablet_bootstrap.cc:492] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:35.074708 25406 ts_tablet_manager.cc:1367] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:35.074934 23893 catalog_manager.cc:5484] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:35.075125 25406 raft_consensus.cc:348] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.075229 25406 raft_consensus.cc:374] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.075273 25406 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.075402 25406 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.075663 25406 ts_tablet_manager.cc:1398] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:35.083752 25283 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.083850 25283 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.084161 25283 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.084491 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:35.084542 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:35.084606 24168 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.084658 24037 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.084868 24232 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.085049 25283 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.085162 25283 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.085237 25283 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.086032 25283 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.086331 25283 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.086670 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:35.086781 24037 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.087121 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:35.087217 24168 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.087612 24037 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.087881 24232 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.087958 24168 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.088106 25283 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.088213 25283 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:35.088346 25283 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.089766 23893 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:35.224118 25283 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.224248 25283 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.224668 25283 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:35.225123 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:35.225121 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:35.225268 24168 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.225268 24430 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.225638 24233 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.225816 25283 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.225876 25283 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.225916 25283 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.226825 25283 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.227166 25283 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:35.227488 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:35.227516 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:35.227583 24168 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.227656 24430 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.228320 24168 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.228404 24430 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.228622 24232 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.228825 25283 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.228902 25283 raft_consensus.cc:686] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:35.229065 25283 consensus_queue.cc:237] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:35.230697 23893 catalog_manager.cc:5484] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
W20240501 07:59:35.361192 25525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.361443 25525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.364656 25525 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 07:59:35.453989 24037 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.454265 24168 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.454538 24167 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.454891 24430 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.454998 25421 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.455320 25421 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.455533 25421 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.455721 25421 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.458626 24037 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.458971 25029 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.459709 24430 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.460153 25335 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.65 ms
time per row: 0.0563249 ms
I20240501 07:59:35.521458 23888 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52876:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:35.527796 24277 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1ffb7f2be64f05ba2fe6277b60f245 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.528026 24278 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.528121 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1ffb7f2be64f05ba2fe6277b60f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.528226 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.530108 24279 tablet_service.cc:1467] Processing CreateTablet for tablet 6523f8bf6c574f36bd31be66abf6fd23 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.530359 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6523f8bf6c574f36bd31be66abf6fd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.530567 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1ffb7f2be64f05ba2fe6277b60f245 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.530865 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1ffb7f2be64f05ba2fe6277b60f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.531150 25409 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:35.531836 25409 tablet_bootstrap.cc:654] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.533001 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.533001 24408 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1ffb7f2be64f05ba2fe6277b60f245 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.533329 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.533510 24408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1ffb7f2be64f05ba2fe6277b60f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.534014 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.534243 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebcc0e2a36945c4ad094dc7a8df62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.535398 25407 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:35.535647 24016 tablet_service.cc:1467] Processing CreateTablet for tablet 6523f8bf6c574f36bd31be66abf6fd23 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.535836 24016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6523f8bf6c574f36bd31be66abf6fd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.536144 25407 tablet_bootstrap.cc:654] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.536381 25408 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:35.537235 25408 tablet_bootstrap.cc:654] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.538370 24146 tablet_service.cc:1467] Processing CreateTablet for tablet 6523f8bf6c574f36bd31be66abf6fd23 (DEFAULT_TABLE table=rebalance_test_table_9 [id=70b061fbaf86404c94296d740073a2f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:35.538643 24146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6523f8bf6c574f36bd31be66abf6fd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:35.540535 25406 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474: Bootstrap starting.
I20240501 07:59:35.541280 25406 tablet_bootstrap.cc:654] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.541474 25409 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:35.541570 25409 ts_tablet_manager.cc:1367] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 07:59:35.542068 25409 raft_consensus.cc:348] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.542193 25409 raft_consensus.cc:374] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.542246 25409 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.542376 25409 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.542668 25409 ts_tablet_manager.cc:1398] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:35.542802 25409 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:35.543704 25409 tablet_bootstrap.cc:654] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.544672 25408 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:35.544757 25408 ts_tablet_manager.cc:1367] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:35.545214 25408 raft_consensus.cc:348] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.545325 25408 raft_consensus.cc:374] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.545369 25408 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.545498 25408 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.545725 25408 ts_tablet_manager.cc:1398] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.545881 25408 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:35.546952 25408 tablet_bootstrap.cc:654] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.548528 25406 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474: No bootstrap required, opened a new log
I20240501 07:59:35.548633 25406 ts_tablet_manager.cc:1367] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:35.549050 25406 raft_consensus.cc:348] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.549151 25406 raft_consensus.cc:374] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.549185 25406 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.549335 25406 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.549626 25408 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:35.549664 25406 ts_tablet_manager.cc:1398] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.549711 25408 ts_tablet_manager.cc:1367] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:35.550060 25408 raft_consensus.cc:348] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.550141 25408 raft_consensus.cc:374] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.550175 25408 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.550294 25408 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.550552 25408 ts_tablet_manager.cc:1398] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.550907 25407 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:35.551046 25407 ts_tablet_manager.cc:1367] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:35.551164 25409 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:35.551265 25409 ts_tablet_manager.cc:1367] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:35.551523 25407 raft_consensus.cc:348] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.551643 25407 raft_consensus.cc:374] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.551661 25408 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f: Bootstrap starting.
I20240501 07:59:35.551687 25407 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.551709 25409 raft_consensus.cc:348] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.551826 25409 raft_consensus.cc:374] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.551858 25409 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.551829 25407 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.551965 25409 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.552163 25407 ts_tablet_manager.cc:1398] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:35.552249 25409 ts_tablet_manager.cc:1398] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:35.552352 25407 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888: Bootstrap starting.
I20240501 07:59:35.552870 25409 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Bootstrap starting.
I20240501 07:59:35.554396 25407 tablet_bootstrap.cc:654] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.554406 25409 tablet_bootstrap.cc:654] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.555608 25407 tablet_bootstrap.cc:492] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888: No bootstrap required, opened a new log
I20240501 07:59:35.555696 25407 ts_tablet_manager.cc:1367] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:35.555755 25409 tablet_bootstrap.cc:492] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: No bootstrap required, opened a new log
I20240501 07:59:35.555838 25409 ts_tablet_manager.cc:1367] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:35.556141 25407 raft_consensus.cc:348] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.556244 25407 raft_consensus.cc:374] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.556221 25409 raft_consensus.cc:348] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.556284 25407 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.556309 25409 raft_consensus.cc:374] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.556339 25409 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.556401 25407 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.556486 25409 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.556711 25407 ts_tablet_manager.cc:1398] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.556790 25409 ts_tablet_manager.cc:1398] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:35.559307 25408 tablet_bootstrap.cc:654] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:35.560529 25408 tablet_bootstrap.cc:492] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f: No bootstrap required, opened a new log
I20240501 07:59:35.560597 25408 ts_tablet_manager.cc:1367] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:35.560964 25408 raft_consensus.cc:348] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.561048 25408 raft_consensus.cc:374] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:35.561069 25408 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Initialized, Role: FOLLOWER
I20240501 07:59:35.561194 25408 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.561441 25408 ts_tablet_manager.cc:1398] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:35.603480 25555 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.603603 25555 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.603986 25555 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:35.604347 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" is_pre_election: true
I20240501 07:59:35.604352 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:35.604470 24035 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.604486 24430 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.604784 24233 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.604998 25555 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.605067 25555 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.605125 25555 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.606081 25555 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.606400 25555 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:35.606724 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:35.606724 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:35.606842 24035 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.606861 24430 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.607635 24430 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.607755 24035 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.607920 24233 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.608119 25555 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.608209 25555 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:35.608358 25555 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.609951 23887 catalog_manager.cc:5484] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:35.689424 25392 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.689540 25392 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.689934 25392 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.690346 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:35.690339 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5" is_pre_election: true
I20240501 07:59:35.690498 24298 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:35.690513 24035 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 0.
I20240501 07:59:35.690801 24366 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.690977 25392 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.691071 25392 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.691099 25392 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.692014 25392 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.692396 25392 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.692714 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:35.692719 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:35.692836 24298 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.692837 24035 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.693742 24298 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:35.693742 24035 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 1.
I20240501 07:59:35.694041 24363 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:35.694247 25392 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.694394 25392 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:35.694555 25392 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:35.696224 23887 catalog_manager.cc:5484] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:35.749086 25555 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:35.749222 25555 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.749528 25555 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.749960 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" is_pre_election: true
I20240501 07:59:35.749996 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474" is_pre_election: true
I20240501 07:59:35.750110 24035 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.750149 24167 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 0.
I20240501 07:59:35.750383 24232 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.750536 25555 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:35.750602 25555 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:35.750656 25555 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.751554 25555 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.751873 25555 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:35.752221 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:35.752242 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:35.752346 24035 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.752357 24167 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:35.753171 24035 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.753366 24167 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 1.
I20240501 07:59:35.753509 24232 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:35.753679 25556 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:35.753769 25556 raft_consensus.cc:686] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:35.753954 25556 consensus_queue.cc:237] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:35.755928 23887 catalog_manager.cc:5484] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:35.879869 25580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.880101 25580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.883141 25580 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 07:59:35.970916 24298 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.971463 25577 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.970978 24035 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.972851 25392 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.978298 24430 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.979089 25556 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.984339 24035 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.984726 25555 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.988181 24168 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.988662 24037 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:35.988968 25555 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:35.989189 25556 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:36.007297 25387 mvcc.cc:205] Tried to move back new op lower bound from 7022798340029276160 to 7022798338493071360. Current Snapshot: MvccSnapshot[applied={T|T < 7022798340000935936}]
INSERT report
rows total: 2000
time total: 116.942 ms
time per row: 0.0584709 ms
I20240501 07:59:36.043956 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:42155
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:46851
--webserver_port=37823
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:36.155712 25625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:36.155946 25625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:36.156030 25625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:36.159248 25625 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:36.159322 25625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:36.159418 25625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:36.163255 25625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:42155
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=37823
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.25625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:36.164151 25625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:36.165272 25625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:59:36.174573 25625 server_base.cc:1034] running on GCE node
W20240501 07:59:36.175596 25633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.176667 25631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.177158 25630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:36.177537 25625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:36.178102 25625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:36.179251 25625 hybrid_clock.cc:648] HybridClock initialized: now 1714550376179238 us; error 27 us; skew 500 ppm
I20240501 07:59:36.181368 25625 webserver.cc:460] Webserver started at http://127.0.99.133:37823/ using document root <none> and password file <none>
I20240501 07:59:36.181918 25625 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:36.182006 25625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:36.185293 25625 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:36.187065 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.188086 25625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:36.188186 25625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fdaa8b9ba4e649359a29a8c224761c46"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:36.188612 25625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:36.204690 25625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:36.205359 25625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:36.205540 25625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:36.206060 25625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:36.206907 25625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:36.206955 25625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.206997 25625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:36.207067 25625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.215917 25625 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42155
I20240501 07:59:36.215945 25752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42155 every 8 connection(s)
I20240501 07:59:36.216957 25625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:36.220546 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 25625
I20240501 07:59:36.220942 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:37171
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:46851
--webserver_port=42529
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:36.226385 25753 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:36.226626 25753 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:36.227222 25753 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:36.228334 23895 ts_manager.cc:194] Re-registered known tserver with Master: fdaa8b9ba4e649359a29a8c224761c46 (127.0.99.133:42155)
I20240501 07:59:36.228870 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39537
W20240501 07:59:36.329715 25756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:36.329949 25756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:36.330009 25756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:36.333252 25756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:36.333317 25756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:36.333402 25756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:36.337251 25756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:37171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=42529
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.25756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:36.338099 25756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:36.339208 25756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:36.347290 25762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.348489 25764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.349519 25761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:36.350458 25756 server_base.cc:1034] running on GCE node
I20240501 07:59:36.350863 25756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:36.351454 25756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:36.352607 25756 hybrid_clock.cc:648] HybridClock initialized: now 1714550376352592 us; error 34 us; skew 500 ppm
I20240501 07:59:36.354596 25756 webserver.cc:460] Webserver started at http://127.0.99.134:42529/ using document root <none> and password file <none>
I20240501 07:59:36.355213 25756 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:36.355294 25756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:36.358888 25756 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:36.360759 25770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.361685 25756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:36.361809 25756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e85aa03a5269488a94c778f62e1b9f06"
format_stamp: "Formatted at 2024-05-01 07:59:26 on dist-test-slave-qm51"
I20240501 07:59:36.362196 25756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:36.373631 25756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:36.374194 25756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:36.374341 25756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:36.374845 25756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:36.375696 25756 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:36.375746 25756 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.375811 25756 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:36.375836 25756 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.384377 25756 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37171
I20240501 07:59:36.384397 25883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37171 every 8 connection(s)
I20240501 07:59:36.385429 25756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:36.386602 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 25756
I20240501 07:59:36.386986 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41365
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:46851
--webserver_port=46723
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:36.395592 25884 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:36.395809 25884 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:36.396262 25884 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:36.397163 23895 ts_manager.cc:194] Re-registered known tserver with Master: e85aa03a5269488a94c778f62e1b9f06 (127.0.99.134:37171)
I20240501 07:59:36.397675 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:58427
W20240501 07:59:36.505517 25887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:36.505745 25887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:36.505812 25887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:36.509127 25887 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:36.509204 25887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:36.509335 25887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:36.513314 25887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41365
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=46723
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.25887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:36.514205 25887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:36.515604 25887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:36.524180 25893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.524992 25892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.526166 25895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:36.528703 25887 server_base.cc:1034] running on GCE node
I20240501 07:59:36.529155 25887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:36.529700 25887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:36.530861 25887 hybrid_clock.cc:648] HybridClock initialized: now 1714550376530847 us; error 26 us; skew 500 ppm
I20240501 07:59:36.537760 25887 webserver.cc:460] Webserver started at http://127.0.99.135:46723/ using document root <none> and password file <none>
I20240501 07:59:36.538341 25887 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:36.538416 25887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:36.541913 25887 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:36.544044 25901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.545033 25887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:36.545125 25887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5069f7538b3145fbaf4ccde56e0f3599"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:36.545496 25887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:36.587865 25887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:36.588537 25887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:36.588706 25887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:36.589254 25887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:36.590235 25887 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:36.590297 25887 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.590358 25887 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:36.590391 25887 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.599020 25887 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41365
I20240501 07:59:36.599068 26014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41365 every 8 connection(s)
I20240501 07:59:36.600003 25887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:36.604347 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 25887
I20240501 07:59:36.605203 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO45e9d/build/debug/bin/kudu
/tmp/dist-test-taskO45e9d/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:35905
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:46851
--webserver_port=34649
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:36.612965 26015 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:36.613188 26015 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:36.613623 26015 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:36.614578 23895 ts_manager.cc:194] Re-registered known tserver with Master: 5069f7538b3145fbaf4ccde56e0f3599 (127.0.99.135:41365)
I20240501 07:59:36.615409 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59873
W20240501 07:59:36.717819 26019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:36.718089 26019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:36.718138 26019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:36.721165 26019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:36.721226 26019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:36.721293 26019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:36.724932 26019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:35905
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=34649
--tserver_master_addrs=127.0.99.190:46851
--never_fsync=true
--heap_profile_path=/tmp/kudu.26019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:36.725776 26019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:36.726825 26019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:36.734973 26027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.735899 26024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:36.736703 26025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:36.738032 26019 server_base.cc:1034] running on GCE node
I20240501 07:59:36.738431 26019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:36.739030 26019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:36.740226 26019 hybrid_clock.cc:648] HybridClock initialized: now 1714550376740200 us; error 38 us; skew 500 ppm
I20240501 07:59:36.742369 26019 webserver.cc:460] Webserver started at http://127.0.99.136:34649/ using document root <none> and password file <none>
I20240501 07:59:36.742918 26019 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:36.742991 26019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:36.746269 26019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:36.748160 26033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.749181 26019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:36.749266 26019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d6ea16b90d4446f4a2f9b4bf61c400fc"
format_stamp: "Formatted at 2024-05-01 07:59:27 on dist-test-slave-qm51"
I20240501 07:59:36.749742 26019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:36.764918 26019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:36.765471 26019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:36.765616 26019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:36.766139 26019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:36.766958 26019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:36.767033 26019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.767100 26019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:36.767125 26019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:36.775820 26019 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:35905
I20240501 07:59:36.775848 26146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:35905 every 8 connection(s)
I20240501 07:59:36.776800 26019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO45e9d/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714550271492631-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:36.782066 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO45e9d/build/debug/bin/kudu as pid 26019
I20240501 07:59:36.782287 398 test_util.cc:268] Using random seed: -567780913
I20240501 07:59:36.786151 26147 heartbeater.cc:347] Connected to a master server at 127.0.99.190:46851
I20240501 07:59:36.786379 26147 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:36.786897 26147 heartbeater.cc:511] Master 127.0.99.190:46851 requested a full tablet report, sending...
I20240501 07:59:36.788270 23895 ts_manager.cc:194] Re-registered known tserver with Master: d6ea16b90d4446f4a2f9b4bf61c400fc (127.0.99.136:35905)
I20240501 07:59:36.788832 23895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:60091
I20240501 07:59:36.793193 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 07:59:36.793386 398 test_util.cc:268] Using random seed: -567769816
I20240501 07:59:36.803440 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 07:59:36.803598 398 test_util.cc:268] Using random seed: -567759603
I20240501 07:59:36.822883 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 07:59:36.823077 398 test_util.cc:268] Using random seed: -567740126
I20240501 07:59:36.836556 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.836733 24037 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.836794 24037 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.837878 24037 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.838280 24037 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.838829 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.838953 24430 raft_consensus.cc:3032] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.838992 24430 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:36.839159 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.839154 24430 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.839264 24035 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.839321 24035 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.839417 24430 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.840198 24430 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.840215 24035 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.840446 23971 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.840510 24035 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.840586 25620 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.840648 25620 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.840814 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.840817 25620 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.840934 24168 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.841308 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.841428 24299 raft_consensus.cc:3032] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.841470 24299 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.841593 24299 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.841838 24299 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.841912 24168 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.842465 23970 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.842629 26018 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.842708 26018 raft_consensus.cc:686] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.842768 24299 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.842872 26018 consensus_queue.cc:237] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.843526 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.843636 24035 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.843693 24035 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.844377 23895 catalog_manager.cc:5484] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.844568 24035 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.844883 24035 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.845554 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.845670 24297 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.845873 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.845961 24298 raft_consensus.cc:3032] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.846019 24298 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.846122 24298 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.846251 24298 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.846247 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.846364 24430 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.846653 24297 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.847230 24298 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.847393 24430 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.848397 23971 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.848651 25620 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.848738 25620 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.848862 25620 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.849066 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.849156 24037 raft_consensus.cc:465] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:36.849797 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.849896 24037 raft_consensus.cc:465] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:36.852347 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.852450 24037 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.852507 24037 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.853427 24037 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.853773 24037 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.854450 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.854553 24037 raft_consensus.cc:465] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:36.854667 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.854782 24430 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.855201 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.855293 24037 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.855340 24037 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.855684 24430 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.855939 23971 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.856225 24037 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.857113 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.857250 24168 raft_consensus.cc:3032] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.857292 24168 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:36.857436 24168 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.857548 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.857579 26018 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.857650 26018 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.857656 24428 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.857704 24168 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.857800 26018 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.858469 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.858539 24428 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.858541 24168 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.858991 23971 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.859176 26018 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.859256 26018 raft_consensus.cc:686] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.859367 26018 consensus_queue.cc:237] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.858564 24167 raft_consensus.cc:3032] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.860294 24167 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:36.860430 24167 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.860584 24167 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.861570 24167 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.863358 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 07:59:36.863543 398 test_util.cc:268] Using random seed: -567699659
I20240501 07:59:36.868337 24037 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.869671 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.869794 24036 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.869845 24036 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:36.869917 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49978: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.870857 24036 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.871258 24036 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.872222 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.872329 24166 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.873418 24166 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.873875 23970 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.874128 25620 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.874212 25620 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.874341 25620 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.874516 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.874603 24036 raft_consensus.cc:465] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:36.876057 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.876175 24298 raft_consensus.cc:3032] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.876219 24298 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.876344 24298 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.876490 24298 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.877199 24430 raft_consensus.cc:1261] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:36.877226 24167 raft_consensus.cc:1261] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:36.877384 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.877491 24036 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.877490 24298 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.877547 24036 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.877569 26213 consensus_queue.cc:1035] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:36.877841 25620 consensus_queue.cc:1035] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:36.878499 24036 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.881670 23890 catalog_manager.cc:5484] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 1 to 2, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.886649 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.886776 24428 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.887945 24428 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.888725 23971 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.888883 25620 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.888969 25620 raft_consensus.cc:686] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.889097 25620 consensus_queue.cc:237] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.890086 24036 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.892611 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.892748 24298 raft_consensus.cc:3032] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.892727 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15ff4003c1654707a17d65993fb938af"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.892802 24298 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.892834 24036 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.892889 24036 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.892935 24298 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.893090 24298 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.893878 24036 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.894224 24036 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.894229 24298 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.895198 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.895310 24428 raft_consensus.cc:3032] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.895349 24428 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:36.895473 24428 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.895625 24428 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.896533 24428 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.896811 23971 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.896966 25620 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.897042 25620 raft_consensus.cc:686] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.897147 25620 consensus_queue.cc:237] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.900056 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.900167 24298 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.901160 24298 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.901582 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.901693 24037 raft_consensus.cc:465] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
W20240501 07:59:36.902649 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57314: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.903635 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.903761 24037 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.903821 24037 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.904752 24037 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.905120 24037 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:36.905894 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.906004 24430 raft_consensus.cc:3032] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.906064 24430 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:36.906193 24430 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.906344 24430 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.906561 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.906674 24037 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.906726 24037 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.907569 24037 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.907910 24037 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.908643 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.908751 24165 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.909780 24165 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.909775 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.909865 24298 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.910717 24298 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.910780 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 07:59:36.910954 398 test_util.cc:268] Using random seed: -567652248
I20240501 07:59:36.911013 23973 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:36.911180 25620 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.911258 25620 raft_consensus.cc:686] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.911376 25620 consensus_queue.cc:237] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.912302 23970 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.912447 26018 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.912518 26018 raft_consensus.cc:686] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.912643 26018 consensus_queue.cc:237] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.913051 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.913152 24037 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.913200 24037 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.914093 24037 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.914206 24430 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
W20240501 07:59:36.914515 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.914603 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.914688 24167 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.915623 24167 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.915882 23970 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.916020 25620 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.916091 25620 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.916188 25620 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.917333 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.917474 24298 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.917518 24298 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.917618 24298 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.917764 24298 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.917743 23895 catalog_manager.cc:5484] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:36.917964 23895 catalog_manager.cc:5484] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:36.918131 23895 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:36.918344 23895 catalog_manager.cc:5484] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.918529 23895 catalog_manager.cc:5484] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.918671 24298 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.918697 23895 catalog_manager.cc:5484] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:36.918957 24037 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.919579 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.918968 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.919684 24036 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.919719 24428 raft_consensus.cc:3032] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.919747 24036 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.919764 24428 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:36.919895 24428 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.920045 24428 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.920579 24036 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.920938 24036 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:36.920957 24428 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.923242 24166 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:36.923722 24429 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:36.924222 25620 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:36.924324 26018 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:36.925773 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.925889 24168 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.926849 24168 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.927206 23970 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.927295 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.927382 25620 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.927410 24036 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.927460 24036 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.927469 25620 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.927596 25620 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.928179 24036 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.928808 24036 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.929016 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.929126 24428 raft_consensus.cc:3032] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.929175 24428 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:36.929313 24428 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.929495 24428 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.929538 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.929646 24166 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.930452 24428 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.930620 24166 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.931432 23893 catalog_manager.cc:5484] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.931577 23971 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.931667 23893 catalog_manager.cc:5484] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:36.931846 23893 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:36.932031 23893 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:36.932096 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.932193 24298 raft_consensus.cc:3032] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.932240 24298 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.932377 24298 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:36.932552 24298 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.932592 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.932703 24035 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.932755 24035 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.933481 25620 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.933553 25620 raft_consensus.cc:686] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.933661 25620 consensus_queue.cc:237] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.933475 24298 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.933668 24035 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.935775 24035 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.936275 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.936314 24036 raft_consensus.cc:1226] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 1. Current term is 2. Ops: [1.4-1.4]
I20240501 07:59:36.936396 24427 raft_consensus.cc:3032] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.936445 24427 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
W20240501 07:59:36.936523 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53208: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.936583 24427 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.936681 24427 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.937634 24427 raft_consensus.cc:2396] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20240501 07:59:36.942490 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.942795 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.942867 24429 raft_consensus.cc:1261] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:36.942925 24297 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.943272 25620 consensus_queue.cc:1035] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:36.944206 24297 raft_consensus.cc:2396] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:36.944639 24297 raft_consensus.cc:1261] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:36.944864 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0272069f81436f81725c3092e0e382"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.944943 23973 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:36.945250 26018 raft_consensus.cc:2726] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:59:36.945322 26213 consensus_queue.cc:1035] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:36.944973 24036 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.945446 24036 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.946400 24036 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.946719 24036 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:36.948417 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.948532 24166 raft_consensus.cc:3032] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.948570 24166 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:36.948694 24166 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.948797 24430 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:36.948868 24166 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.949146 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.949173 25620 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:36.949247 24427 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.949880 24166 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.950261 24427 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.950299 23970 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:36.950459 25620 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.950536 25620 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.950657 25620 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.952329 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_5 already exists
I20240501 07:59:36.952509 398 test_util.cc:268] Using random seed: -567610694
I20240501 07:59:36.955905 24298 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:36.956554 23890 catalog_manager.cc:5484] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:36.956863 23890 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:36.958540 26213 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:36.958690 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.958787 24036 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.958832 24036 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.959704 24036 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.960065 24036 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.961531 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:36.961655 24165 raft_consensus.cc:3032] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.961706 24165 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:36.961848 24165 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.961954 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.962041 24430 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.962038 24165 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.962656 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.962786 24036 raft_consensus.cc:465] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:36.962918 24165 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.962918 24430 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.963225 23971 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.963551 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:36.963652 24036 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.963704 24036 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.964123 26213 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.964202 26213 raft_consensus.cc:686] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.964313 26213 consensus_queue.cc:237] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.965541 24036 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.965900 24036 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:36.967293 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.967406 24430 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.968178 23890 catalog_manager.cc:5484] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:36.968333 24430 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
I20240501 07:59:36.968969 23971 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:36.969128 26018 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:36.969203 26018 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:36.969339 26018 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.977993 23890 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:36.981896 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:36.982028 24297 raft_consensus.cc:3032] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:36.982070 24297 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.982199 24297 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:36.982404 24297 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:36.983464 24297 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 2.
W20240501 07:59:36.985023 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:36.988318 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:36.988473 24299 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.988546 24299 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:36.989605 24299 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:36.990016 24299 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:36.993191 24298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:36.993304 24298 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.993357 24298 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:36.994441 24298 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.994774 24298 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.995308 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:36.995451 24428 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:36.996140 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:36.996251 24296 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:36.996305 24296 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:36.996510 24428 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:36.996825 24233 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:36.997030 25421 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:36.997119 25421 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:36.997264 25421 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.997249 24296 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:36.997893 24296 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:36.998904 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.000432 24428 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.000509 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.000627 24036 raft_consensus.cc:3032] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.000672 24036 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.000828 24036 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.001096 24036 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.001447 24428 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.001758 24233 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.001955 25421 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.002024 25421 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.002144 25421 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.002054 24036 raft_consensus.cc:2396] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.003742 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.003852 24035 raft_consensus.cc:3032] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.003894 24035 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.004034 24035 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.004187 24035 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.004452 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.004537 24036 raft_consensus.cc:3032] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.004575 24036 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.004745 24036 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.004892 24036 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.005158 24035 raft_consensus.cc:2396] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.005193 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.005295 24299 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.005347 24299 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.006244 24036 raft_consensus.cc:2396] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.006278 24299 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.006614 24299 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.006914 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.006929 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.007059 24167 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.007095 24428 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.007704 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.007822 24299 raft_consensus.cc:465] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:37.008085 24167 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.008394 24428 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.008688 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.008788 24036 raft_consensus.cc:3032] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.008827 24036 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.008828 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.008913 24299 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.008960 24299 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.008975 24036 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.009140 24036 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.009786 24299 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.009896 24036 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.010146 24232 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.010298 25556 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.010368 25556 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.010490 25556 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.010636 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.011205 24429 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.012005 24232 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters: 1df5867ab7af4bd0a8229838d80e9d7f
I20240501 07:59:37.012226 24429 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.012281 25421 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.012372 25421 raft_consensus.cc:686] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.012499 25421 consensus_queue.cc:237] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.012732 24233 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.012935 25556 raft_consensus.cc:2781] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.013038 25556 raft_consensus.cc:686] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.013154 25556 consensus_queue.cc:237] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.015065 23890 catalog_manager.cc:5484] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.016132 24299 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.016315 24428 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.016636 26213 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.017050 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.017153 24299 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.017208 24299 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.018088 24299 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.018618 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.018716 24035 raft_consensus.cc:3032] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.018764 24035 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.018873 24035 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.018960 24035 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.019851 24035 raft_consensus.cc:2396] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 2 because replica has last-logged OpId of term: 1 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240501 07:59:37.020486 24297 raft_consensus.cc:1226] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 1. Current term is 2. Ops: [1.8-1.8]
I20240501 07:59:37.020581 24167 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.021437 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.021528 24036 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.022425 24299 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.022536 24036 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.023095 26213 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.023169 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.027045 24299 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.027130 24299 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.028405 24299 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.029151 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.029268 24167 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.029836 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.029958 24166 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.030987 24166 raft_consensus.cc:2396] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 2 because replica has last-logged OpId of term: 1 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240501 07:59:37.031430 24167 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.032754 24232 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d371a71c02a4432aca2fed33b2545d5; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.032999 25421 raft_consensus.cc:2726] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:59:37.033066 24232 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.033270 26210 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.033355 26210 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.033481 26210 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.034673 23890 catalog_manager.cc:5484] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 3, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.039420 24299 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.039685 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.039808 24430 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.039855 24430 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.039979 24430 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.040095 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.040158 24430 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.040196 24299 raft_consensus.cc:465] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:37.040669 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.040773 24299 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.040827 24299 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.034965 23890 catalog_manager.cc:5484] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.041174 24430 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.041134 23890 catalog_manager.cc:5484] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.041307 23890 catalog_manager.cc:5484] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.041731 24299 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.042735 24297 raft_consensus.cc:1226] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 2. Current term is 3. Ops: [2.8-2.8]
I20240501 07:59:37.043794 24299 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.044323 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.044410 24297 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.044405 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.044464 24297 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.044512 24428 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.045425 24297 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.045568 24428 raft_consensus.cc:2396] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240501 07:59:37.045758 24297 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.046316 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.046437 24165 raft_consensus.cc:3032] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.046480 24165 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.046615 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.046619 24165 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.046710 24297 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.046770 24297 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.046816 24165 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.047253 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.047353 24036 raft_consensus.cc:3032] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.047387 24036 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.047673 24297 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.047946 24165 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.048281 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.048326 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.048391 24427 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.048429 24428 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.049119 26018 consensus_queue.cc:1046] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: INVALID_TERM, Last received: 2.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.049201 24428 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.049322 24427 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.049521 24297 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.049551 24233 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.049758 24233 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.049912 26210 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.049998 26210 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.050148 26210 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.050662 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.050781 24035 raft_consensus.cc:3032] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.050832 24035 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.050827 26210 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.050905 26210 raft_consensus.cc:686] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.050959 24035 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.051054 26210 consensus_queue.cc:237] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.051144 24035 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.051370 24036 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.051525 24036 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.052065 24035 raft_consensus.cc:2396] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
I20240501 07:59:37.052961 23890 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.053826 24298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15ff4003c1654707a17d65993fb938af"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.053932 24298 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.053984 24298 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.054832 24298 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.055195 24298 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.055578 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.055689 24428 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.055755 24036 raft_consensus.cc:2396] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
W20240501 07:59:37.055835 26018 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240501 07:59:37.056609 24428 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.056614 24297 raft_consensus.cc:1226] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 2. Current term is 3. Ops: [2.10-2.10]
I20240501 07:59:37.056887 24233 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.057606 25421 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.058491 25421 raft_consensus.cc:686] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.058645 25421 consensus_queue.cc:237] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.059340 23890 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.059581 23890 catalog_manager.cc:5484] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.060024 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.060040 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.060135 24035 raft_consensus.cc:3032] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.060135 24299 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.060184 24035 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.060196 24299 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.060328 24035 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.060462 24035 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.061065 24299 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.061415 24035 raft_consensus.cc:2396] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.061842 24299 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.061904 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.062023 24165 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.062345 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.062436 24297 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.062485 24297 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.063112 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.063206 24035 raft_consensus.cc:3032] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.063241 24035 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.063340 24165 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.063360 24035 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.063509 24035 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.063817 24232 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.064051 25421 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.064218 25421 raft_consensus.cc:686] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.064363 25421 consensus_queue.cc:237] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.064476 24035 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
W20240501 07:59:37.065105 26169 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.168s ago.
I20240501 07:59:37.065691 23890 catalog_manager.cc:5484] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.064064 24297 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.066570 24297 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.067143 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.067237 24297 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.067209 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.067282 24297 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.067301 24167 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.068050 24297 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.068109 24167 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.068372 24297 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.068917 24232 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d371a71c02a4432aca2fed33b2545d5; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.068969 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21809c44bb940ffb007041977ac5696"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.069068 24297 raft_consensus.cc:465] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:37.069345 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.069438 24167 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.069600 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.069700 24299 raft_consensus.cc:465] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 07:59:37.070273 24167 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.070262 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.070371 24299 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.070438 24299 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.070601 25421 raft_consensus.cc:2726] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 07:59:37.070756 24232 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.070904 24232 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.071094 25556 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.071179 25556 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.071326 25556 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.071434 24299 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.071759 24299 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.072077 25556 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.072165 25556 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.072279 25556 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
W20240501 07:59:37.073690 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.073961 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.074054 24297 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.074041 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.074102 24297 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.074136 24167 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.074889 24167 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.074891 24297 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.075234 24297 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.075682 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.075791 24167 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.075927 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.076042 24430 raft_consensus.cc:3032] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.076089 24430 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.076248 24430 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.076426 24430 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.076632 24167 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.076707 23890 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.076930 24232 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.077085 24232 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.077234 26210 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.077127 23890 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.077324 26210 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.077411 24430 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.077458 26210 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.077602 23890 catalog_manager.cc:5484] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.077703 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.077805 24299 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.077865 24299 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.078063 26210 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.078130 26210 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.078233 26210 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.078684 24299 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.079028 24299 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.079105 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.079208 24035 raft_consensus.cc:3032] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.079252 24035 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.079628 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.079667 24035 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.079720 24299 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.079774 24299 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.079823 24035 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.080531 24299 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.081080 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.081179 24428 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.081871 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.081991 24037 raft_consensus.cc:3032] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.082032 24037 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.082059 24428 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 3.
I20240501 07:59:37.082204 24037 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.082357 24037 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.082366 24233 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.082516 25421 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.082635 25421 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.082796 25421 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.083359 24037 raft_consensus.cc:2396] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.083784 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.083940 24167 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.084154 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.084262 24037 raft_consensus.cc:3032] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.084303 24037 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.084455 24037 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.084551 24299 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 3 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.084633 24037 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.084826 24167 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.085175 24232 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.085314 25421 raft_consensus.cc:2781] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.085423 25421 raft_consensus.cc:686] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.085541 24037 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.085534 25421 consensus_queue.cc:237] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.086174 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:37.086248 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_6 already exists
I20240501 07:59:37.086261 24297 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.086316 24297 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.086402 398 test_util.cc:268] Using random seed: -567476800
I20240501 07:59:37.087285 24035 raft_consensus.cc:2396] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240501 07:59:37.087590 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.087687 24036 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.087738 24036 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.087886 24036 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.088043 24036 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.088587 24430 raft_consensus.cc:1261] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.088912 24036 raft_consensus.cc:2396] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.088950 25421 consensus_queue.cc:1035] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.089295 23890 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.090456 24297 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.090791 24297 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.091334 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.091428 24167 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.092669 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.092693 24167 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.092770 24037 raft_consensus.cc:3032] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.092810 24037 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.092949 24232 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.093055 24037 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.093210 24037 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.093402 25421 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.093484 25421 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.093607 25421 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.094800 24037 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 2.
I20240501 07:59:37.092530 23890 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.095499 23890 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.096302 24035 raft_consensus.cc:1261] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.097939 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.098060 24036 raft_consensus.cc:3032] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.098140 24036 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
W20240501 07:59:37.098603 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53240: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.098282 24036 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.099054 24036 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.099457 23893 catalog_manager.cc:5484] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.099634 23893 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 1 to 2, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.099972 24036 raft_consensus.cc:2396] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.101255 25556 consensus_queue.cc:1035] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.101369 25556 consensus_queue.cc:1225] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer 1df5867ab7af4bd0a8229838d80e9d7f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.102619 26204 mvcc.cc:205] Tried to move back new op lower bound from 7022798344545910784 to 7022798344433577984. Current Snapshot: MvccSnapshot[applied={T|T < 7022798344545910784}]
I20240501 07:59:37.105633 24036 pending_rounds.cc:76] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting all ops after (but not including) 3
I20240501 07:59:37.105785 24036 pending_rounds.cc:97] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.105929 24036 raft_consensus.cc:2866] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.111308 24427 raft_consensus.cc:1261] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.111625 24165 raft_consensus.cc:1261] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.111815 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.111955 24166 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.112023 24166 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.112004 25556 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.113109 24166 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.113535 24166 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.114362 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.114485 24035 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.114892 25556 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.115550 24035 raft_consensus.cc:2396] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.118541 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.118664 24166 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.118716 24166 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.119678 24166 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.120061 24166 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.121434 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.121562 24037 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.122530 24037 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.123760 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.123875 24165 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.123936 24165 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.124053 24101 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:37.124224 26194 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.124290 26194 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.124408 26194 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.124876 24165 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.125250 24165 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.127202 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_7 already exists
I20240501 07:59:37.127351 398 test_util.cc:268] Using random seed: -567435850
I20240501 07:59:37.127447 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.127557 24165 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.127607 24165 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.127564 23893 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.128391 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.128453 24165 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.128509 24298 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.129016 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.129117 24429 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.129511 24298 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.129961 24104 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.130085 24429 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.130157 25335 raft_consensus.cc:2781] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.130234 25335 raft_consensus.cc:686] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.130362 24102 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.130362 25335 consensus_queue.cc:237] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.130652 24165 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.130833 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.130920 25335 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.130939 24037 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.130986 25335 raft_consensus.cc:686] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.131156 25335 consensus_queue.cc:237] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.131877 24037 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
W20240501 07:59:37.132813 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.133267 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.133355 24165 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.133406 24165 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.134275 24165 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.134305 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.134408 24298 raft_consensus.cc:3032] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.134440 24298 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.134558 24298 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.134581 24165 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.134709 24298 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.134780 23890 catalog_manager.cc:5484] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.134999 23890 catalog_manager.cc:5484] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.135660 24298 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.136219 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.136320 24429 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.137239 24429 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.137522 24102 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.138119 26194 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.138212 26194 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.138339 26194 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:37.142588 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.142861 24167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.142956 24167 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.143041 24167 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.143431 24429 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.143772 26210 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.143993 24167 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.144320 24167 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.144623 24165 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.145133 26210 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.145506 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.145594 24428 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.146485 24428 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.147572 24102 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.147967 25335 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.148047 25335 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.148056 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.148154 24298 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.148188 24298 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.148164 25335 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.148319 24298 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.148450 24298 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:37.149109 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.150107 23893 catalog_manager.cc:5484] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.152417 24167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.152541 24167 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.152597 24167 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.153501 24167 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.154330 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.154431 24429 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.154500 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.154615 24296 raft_consensus.cc:3032] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.154652 24296 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.154793 24296 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.154958 24296 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.155318 24429 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.155598 24102 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.155781 26194 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.155858 26194 raft_consensus.cc:686] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.155969 26194 consensus_queue.cc:237] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.156309 23893 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.156797 24167 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.157322 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.157423 24166 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.157474 24166 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:37.157853 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.155696 24298 raft_consensus.cc:2396] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.158365 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.158375 24166 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.158464 24036 raft_consensus.cc:3032] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.158499 24036 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.158635 24296 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.158620 24036 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.158700 24166 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.158802 24036 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.159768 24036 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.160326 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.160445 24428 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.161314 24428 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.161617 24102 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.161731 26194 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.161809 26194 raft_consensus.cc:686] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.161922 26194 consensus_queue.cc:237] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.162293 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21809c44bb940ffb007041977ac5696"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.162392 24166 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.162438 24166 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.163491 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.163574 24035 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.163967 24296 raft_consensus.cc:3032] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.163972 24166 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.164016 24296 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.163904 23893 catalog_manager.cc:5484] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.164142 24296 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.164269 24296 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.164275 24166 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.164812 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.164916 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.165002 24430 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.165208 24296 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.165661 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.165779 24297 raft_consensus.cc:3032] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.165894 24297 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.165989 24430 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.166115 24297 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.166291 24297 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.166814 24102 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.166961 26194 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.167074 26194 consensus_queue.cc:1225] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.167318 26194 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.167341 24297 raft_consensus.cc:2396] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.167385 26194 raft_consensus.cc:686] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.167493 26194 consensus_queue.cc:237] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.168346 24104 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.168561 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.168643 24298 raft_consensus.cc:3032] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.168673 24298 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.168790 24298 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.168841 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_8 already exists
I20240501 07:59:37.168920 24298 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.169000 398 test_util.cc:268] Using random seed: -567394201
I20240501 07:59:37.169701 24298 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.163585 24036 raft_consensus.cc:3032] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.170192 24036 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.170325 24036 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.170477 24036 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.171341 24036 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
W20240501 07:59:37.171907 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.172582 26309 raft_consensus.cc:2726] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.175947 26309 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.177116 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.177223 24165 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.177276 24165 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.178293 24165 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.178632 24165 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
W20240501 07:59:37.179414 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.180049 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57206: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.180128 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.180227 24427 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.181280 24427 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.181546 24102 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.182252 26194 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.182248 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.182369 26194 raft_consensus.cc:686] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.182382 24165 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.182427 24165 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.182502 26194 consensus_queue.cc:237] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.183342 24165 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.183393 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.183493 24298 raft_consensus.cc:3032] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.183530 24298 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.183679 24165 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.183686 24298 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.183923 24298 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.184115 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.184197 24296 raft_consensus.cc:3032] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.184227 24296 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.184334 24296 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.184442 24296 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.184893 24298 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.185508 23890 catalog_manager.cc:5484] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.185757 23890 catalog_manager.cc:5484] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.185846 24296 raft_consensus.cc:2396] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.186257 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.186363 24035 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.186408 26273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: Replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.188565 24035 raft_consensus.cc:2396] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20240501 07:59:37.189150 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.189653 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.189754 24165 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.189807 24165 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.190646 24101 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.190862 25335 raft_consensus.cc:2726] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.191395 24165 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.191476 24297 pending_rounds.cc:76] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.191520 24297 pending_rounds.cc:97] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.191578 24297 raft_consensus.cc:2866] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.191769 24165 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.191993 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.192080 24296 raft_consensus.cc:3032] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.192109 24296 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.192222 24296 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.192346 24296 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.192503 23890 catalog_manager.cc:5484] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 2, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
W20240501 07:59:37.192941 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50010: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.193189 24296 raft_consensus.cc:2396] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 3 because replica has last-logged OpId of term: 2 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 12.
I20240501 07:59:37.193886 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.193991 24035 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.194978 24035 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
W20240501 07:59:37.194976 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.195489 24101 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.195899 25335 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.195981 25335 raft_consensus.cc:686] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.196130 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.196105 25335 consensus_queue.cc:237] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.196205 24165 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.196247 24165 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.197520 24165 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.197803 24165 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.198817 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.198927 24035 raft_consensus.cc:3032] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.198967 24035 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.199139 24035 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.199287 24035 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.198998 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.199849 24430 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.200155 24035 raft_consensus.cc:2396] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.200675 24037 raft_consensus.cc:1261] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.200850 24430 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.201004 24166 raft_consensus.cc:1261] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.201035 26210 consensus_queue.cc:1035] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.201494 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.201664 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.202551 26210 consensus_queue.cc:1035] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.202598 24035 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.202772 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.202869 24165 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.202922 24165 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.203292 26210 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.203375 26210 consensus_queue.cc:1225] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer 1df5867ab7af4bd0a8229838d80e9d7f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.204187 24165 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.204398 24102 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.204428 24164 raft_consensus.cc:1226] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Rejecting Update request from peer 9d371a71c02a4432aca2fed33b2545d5 for earlier term 2. Current term is 3. Ops: [2.4-2.5]
I20240501 07:59:37.205430 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.205536 24296 raft_consensus.cc:3032] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.205571 24296 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.205701 24296 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.205811 24296 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.206532 24035 pending_rounds.cc:76] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting all ops after (but not including) 3
I20240501 07:59:37.206581 24035 pending_rounds.cc:97] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.206619 24035 raft_consensus.cc:2866] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.206772 24296 raft_consensus.cc:2396] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.207118 24165 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.208022 26194 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.208106 26194 raft_consensus.cc:686] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.208977 26194 consensus_queue.cc:237] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.209559 24037 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.209728 26273 mvcc.cc:205] Tried to move back new op lower bound from 7022798344999047168 to 7022798344486043648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798344999047168 or (T in {7022798344999047168})}]
I20240501 07:59:37.209861 24167 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.210177 25556 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.210393 25556 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.211639 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.212291 24430 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.212635 26210 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.212798 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_9 already exists
W20240501 07:59:37.216233 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.217345 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.218431 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.218533 24165 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.218583 24165 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:37.219627 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57358: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.219635 24165 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.219951 24165 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.220242 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.220343 24427 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.220695 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.220794 24036 raft_consensus.cc:3032] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.220830 24036 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.220953 24036 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.221130 24036 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.221230 24427 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.221340 23890 catalog_manager.cc:5484] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.221463 24102 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.221604 23890 catalog_manager.cc:5484] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.221663 26194 raft_consensus.cc:2781] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.221729 26194 raft_consensus.cc:686] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.221840 26194 consensus_queue.cc:237] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.222110 24036 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 2.
I20240501 07:59:37.223388 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.223506 24035 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.224261 24430 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.224378 24035 raft_consensus.cc:2396] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.224690 26271 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.227878 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.227908 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57374: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.228536 24167 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.228886 26213 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.229899 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.231197 25753 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:37.232573 24296 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.232930 26309 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.233026 26309 consensus_queue.cc:1225] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.233530 24296 pending_rounds.cc:76] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.233567 24296 pending_rounds.cc:97] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.233549 24427 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.233593 24296 raft_consensus.cc:2866] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.234062 26309 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.235029 26177 mvcc.cc:205] Tried to move back new op lower bound from 7022798345137680384 to 7022798344800415744. Current Snapshot: MvccSnapshot[applied={T|T < 7022798345137680384}]
I20240501 07:59:37.236675 24101 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.236840 26194 raft_consensus.cc:2726] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 07:59:37.237068 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.237160 24165 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.237206 24165 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.237677 23893 catalog_manager.cc:5484] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 1 to 2, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.238281 24165 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.238579 24165 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.239293 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.239393 24428 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:37.239794 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.240080 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.240182 24035 raft_consensus.cc:3032] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.240218 24035 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.240350 24428 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.240350 24035 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.240494 24035 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.240630 24102 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.240782 26194 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.240850 26194 raft_consensus.cc:686] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.240999 26194 consensus_queue.cc:237] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.241362 24035 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.243674 26263 mvcc.cc:205] Tried to move back new op lower bound from 7022798345103437824 to 7022798343921807360. Current Snapshot: MvccSnapshot[applied={T|T < 7022798345103437824}]
I20240501 07:59:37.243968 24427 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.244321 24296 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.244421 26194 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.245834 26309 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.246044 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.246145 24165 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.246201 24165 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.247174 24165 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.247483 24165 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.248603 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.248718 24035 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.249591 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.249689 24165 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.249699 24035 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 4.
I20240501 07:59:37.249749 24165 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.250159 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.250257 24298 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.250293 24298 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.250460 24298 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.250641 24298 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.250620 24165 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.250939 24165 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.251878 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.251991 24299 raft_consensus.cc:3032] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.252035 24299 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.252163 24299 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.252282 24299 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.253070 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.254459 24299 raft_consensus.cc:2396] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240501 07:59:37.256287 24036 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.256732 24428 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.256853 24298 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 4.
I20240501 07:59:37.257232 24104 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.257308 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.257422 24036 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.258325 26194 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.258383 24036 raft_consensus.cc:2396] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240501 07:59:37.258423 26194 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.258553 26194 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.259068 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.259546 24101 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.259544 26299 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.259737 26309 raft_consensus.cc:2726] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.259929 26299 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.260026 26299 consensus_queue.cc:1225] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer 1df5867ab7af4bd0a8229838d80e9d7f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.260169 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.260272 24165 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.260324 24165 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.261551 24165 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.261899 24036 pending_rounds.cc:76] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting all ops after (but not including) 3
I20240501 07:59:37.261955 24036 pending_rounds.cc:97] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.262002 24036 raft_consensus.cc:2866] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.262179 24165 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.262523 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.262636 24428 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.263540 24428 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.263835 24102 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.264006 26309 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.263902 23893 catalog_manager.cc:5484] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.264060 26309 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.264189 26309 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.264858 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.265010 24165 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.265115 24165 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.266125 24165 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.266499 24165 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.267529 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.267647 24296 raft_consensus.cc:3032] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.267699 24296 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.267829 24296 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.267968 24296 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.269006 24296 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.269407 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.269510 24296 raft_consensus.cc:3032] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.269558 24296 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
W20240501 07:59:37.269675 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.269694 24296 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.269842 24296 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.270692 24296 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.271343 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.271461 24428 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.272312 24428 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.272709 24102 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.273273 26194 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.273358 26194 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.273506 26194 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
W20240501 07:59:37.275970 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.283447 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57358: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.284610 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.285887 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.286865 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.288090 24296 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240501 07:59:37.289819 26194 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:37.291085 24036 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240501 07:59:37.292006 24427 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.292378 26271 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.292896 24167 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:37.293339 24167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.293434 24167 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.293498 24167 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.293704 26271 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.294600 24167 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.294567 23890 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 3 to 4, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.294925 25335 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:37.295095 24167 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.294946 23890 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.295341 23890 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:37.297128 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.298084 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.298202 24429 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.299278 24429 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
W20240501 07:59:37.299384 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.299628 24102 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.299800 25335 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.299891 25335 raft_consensus.cc:686] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.300050 25335 consensus_queue.cc:237] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:37.305768 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.307391 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.307528 24036 raft_consensus.cc:3032] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.307566 24036 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.307704 24036 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.307880 24036 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.308373 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0272069f81436f81725c3092e0e382"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:37.308480 24165 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.308537 24165 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.308912 24036 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.309470 24165 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.309777 24165 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.310562 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.311719 24165 raft_consensus.cc:1226] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 2. Current term is 3. Ops: []
W20240501 07:59:37.311735 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.311693 23895 catalog_manager.cc:5484] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.312678 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.312784 24035 raft_consensus.cc:3032] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.312824 24035 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.312958 24035 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.313068 26213 consensus_queue.cc:1159] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3748d2c0aeb740f98f8c2caabad21474" responder_term: 3 status { last_received { term: 2 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 07:59:37.313732 24035 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.314638 24035 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 3.
I20240501 07:59:37.316565 24101 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters: 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.316802 26340 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.316967 26340 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.317111 26340 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.319095 23895 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 2 to 3, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:37.323853 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.325824 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.326037 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.330621 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.330785 24430 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.330858 24430 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.331918 24430 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.332290 24430 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.333261 24428 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.333470 24037 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.333681 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.333698 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.333771 24430 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.333806 24035 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.333818 24430 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.334563 24035 raft_consensus.cc:2396] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.334630 24430 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.334867 25335 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.334954 24430 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.335677 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.335788 24037 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.335886 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.335974 24430 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.336017 24430 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.336769 24430 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.336930 24037 raft_consensus.cc:2396] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.337062 24430 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.339529 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
W20240501 07:59:37.339839 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.340142 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.340262 24165 raft_consensus.cc:3032] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.340312 24165 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.340447 24165 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.339613 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
W20240501 07:59:37.340179 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.340621 24165 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.340901 24296 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:37.341323 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.341508 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.341609 24429 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.341653 24429 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.341953 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.342053 24297 raft_consensus.cc:3032] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.342092 24297 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.342301 24165 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.342501 24429 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.342506 24297 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.342654 24297 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.342849 24429 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.343724 24296 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.343727 24363 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.343685 26341 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.343811 26341 consensus_queue.cc:1225] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Peer 1df5867ab7af4bd0a8229838d80e9d7f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.343793 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.343890 26221 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.343890 24165 raft_consensus.cc:3032] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.343948 24165 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.343966 26221 raft_consensus.cc:686] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.344015 24298 raft_consensus.cc:3032] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.344067 24298 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.344103 24165 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.14, Last appended by leader: 14, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.344105 26221 consensus_queue.cc:237] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.344230 24165 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.344251 24298 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.344386 24298 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.344430 24035 pending_rounds.cc:76] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting all ops after (but not including) 3
I20240501 07:59:37.344468 24035 pending_rounds.cc:97] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.344498 24035 raft_consensus.cc:2866] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.345719 24165 raft_consensus.cc:2396] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 3 because replica has last-logged OpId of term: 2 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 13.
W20240501 07:59:37.345957 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.346019 24298 raft_consensus.cc:2396] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.345994 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.346115 24429 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.346166 24429 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.346314 24297 raft_consensus.cc:2396] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20240501 07:59:37.347136 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.347769 24429 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.348079 24429 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.350270 24366 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.350476 25392 raft_consensus.cc:2726] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.350919 24366 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.351069 25392 raft_consensus.cc:2726] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.351641 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.351763 24035 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.351836 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.351931 24429 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.351982 24429 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.353010 24429 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.353358 24035 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.353590 24429 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
W20240501 07:59:37.353706 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.354156 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.354257 24165 raft_consensus.cc:3032] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.354292 24165 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.354380 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.354415 24165 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.354485 24035 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.354563 24165 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.355399 24035 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.355532 24165 raft_consensus.cc:2396] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11.
I20240501 07:59:37.356215 24363 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.356352 25392 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.356410 25392 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.356520 25392 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.357235 24363 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.357236 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
W20240501 07:59:37.357384 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.357395 26221 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.357488 26221 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.357537 24429 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.357606 24429 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.357601 26221 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.358120 24035 raft_consensus.cc:1226] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Rejecting Update request from peer 9d371a71c02a4432aca2fed33b2545d5 for earlier term 2. Current term is 3. Ops: [2.8-2.8]
I20240501 07:59:37.358429 24429 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.358922 26299 consensus_queue.cc:1046] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: INVALID_TERM, Last received: 2.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.358911 23890 catalog_manager.cc:5484] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.359130 26299 raft_consensus.cc:3032] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.359190 26299 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
W20240501 07:59:37.359315 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.359362 26299 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.361531 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.362354 24427 raft_consensus.cc:1226] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Rejecting Update request from peer 9d371a71c02a4432aca2fed33b2545d5 for earlier term 2. Current term is 3. Ops: [2.8-2.8]
I20240501 07:59:37.362643 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.362792 24296 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.363786 24296 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.363893 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.364219 24366 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.364421 25392 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.364477 25392 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.364573 25392 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.365350 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.365911 24297 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.366451 25421 consensus_queue.cc:1046] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: INVALID_TERM, Last received: 2.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.366621 25421 consensus_queue.cc:1225] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Peer 3b8f81da8ae74d3687ef72a2c9ce0888 log is divergent from this leader: its last log entry 3.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240501 07:59:37.366791 26299 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:37.367277 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.367956 24298 raft_consensus.cc:2396] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 3 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
W20240501 07:59:37.369585 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.370050 24037 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 07:59:37.370435 24429 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.370774 26341 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:37.374882 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.376294 24167 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.376391 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.376505 24165 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.376551 24165 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.376698 24165 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.376883 24165 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.376931 24429 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.377102 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.377195 24430 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.377239 24430 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.377940 24429 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 07:59:37.378015 24298 raft_consensus.cc:1261] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.378240 24165 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.378221 26221 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.378739 24430 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.378926 26340 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.379595 24297 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.379906 26221 consensus_queue.cc:1035] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:37.380146 24366 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters: 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.382808 26221 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.382884 26221 raft_consensus.cc:686] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.382992 26221 consensus_queue.cc:237] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.384680 24430 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.385510 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.385648 24298 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.387638 24298 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.388386 24366 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.388550 26221 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.388615 26221 raft_consensus.cc:686] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
W20240501 07:59:37.388710 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.388739 26221 consensus_queue.cc:237] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:37.389477 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.389542 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.391086 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.391192 24165 raft_consensus.cc:3032] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.391242 24165 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.391348 24165 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.391464 24165 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.392396 24165 raft_consensus.cc:2396] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 3 because replica has last-logged OpId of term: 2 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 7.
I20240501 07:59:37.392827 23890 catalog_manager.cc:5484] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.393285 23890 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.393474 23890 catalog_manager.cc:5484] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:37.394299 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.395565 24037 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 07:59:37.396111 26340 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:37.399499 25884 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:37.400045 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
W20240501 07:59:37.400960 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.408067 24429 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.408182 24429 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.413653 24428 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
W20240501 07:59:37.413842 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.414132 24429 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:37.417766 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.418440 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.424480 24429 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.425171 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.425287 24036 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.425745 26309 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:37.425843 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.425921 24167 raft_consensus.cc:3032] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.425956 24167 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.426067 24167 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.426191 24167 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.426321 24036 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.427269 24167 raft_consensus.cc:2396] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240501 07:59:37.432097 23895 catalog_manager.cc:5484] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:37.433439 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.435163 24363 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.435389 25392 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.435472 25392 raft_consensus.cc:686] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.435622 25392 consensus_queue.cc:237] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.436822 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21809c44bb940ffb007041977ac5696"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.436944 24430 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.437012 24430 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.438388 24430 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:37.439319 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.440032 24430 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.440606 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.440738 24297 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.440891 23895 catalog_manager.cc:5484] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.441627 24297 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
W20240501 07:59:37.443248 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.444000 24366 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.444171 25392 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.444260 25392 raft_consensus.cc:686] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.444401 25392 consensus_queue.cc:237] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:37.447732 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.450907 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.451048 24165 raft_consensus.cc:3032] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.451089 24165 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.451208 24165 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.451316 24165 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.452381 24165 raft_consensus.cc:2396] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:37.458137 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.458254 24428 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.458310 24428 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.459430 24428 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.459794 24428 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.460875 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.460999 24297 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.462045 24297 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.463721 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.464216 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.467559 24366 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.467746 26221 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.467828 26221 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.467975 26221 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
W20240501 07:59:37.469342 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.474788 23895 catalog_manager.cc:5484] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.475440 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.475554 24165 raft_consensus.cc:3032] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.475593 24165 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.475736 24165 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.475885 24165 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.476632 23895 catalog_manager.cc:5484] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.476967 24165 raft_consensus.cc:2396] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
W20240501 07:59:37.485172 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.487632 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.489727 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.489856 24428 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.489909 24428 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.490953 24428 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.491322 24428 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
W20240501 07:59:37.491926 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57350: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.492106 24037 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.492609 25421 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.495276 23895 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.496176 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.496284 24298 raft_consensus.cc:3032] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.496326 24298 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.496449 24298 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.496588 24298 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.497292 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.497387 24428 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.497447 24428 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.497671 24298 raft_consensus.cc:2396] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
I20240501 07:59:37.497689 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.497802 24035 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.498399 24428 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.498768 24428 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.498847 24035 raft_consensus.cc:2396] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 2 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
I20240501 07:59:37.500785 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.500890 24035 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.501221 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.501330 24167 raft_consensus.cc:3032] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:37.501375 24167 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.501499 24167 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.501646 24167 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:37.501848 24035 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 3.
I20240501 07:59:37.502676 24167 raft_consensus.cc:2396] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 3 because replica has last-logged OpId of term: 2 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 9.
W20240501 07:59:37.503691 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.503729 24363 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.503991 24363 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.504074 26221 raft_consensus.cc:2726] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.504338 26347 raft_consensus.cc:2781] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:37.504432 26347 raft_consensus.cc:686] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.504549 26347 consensus_queue.cc:237] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.506181 24429 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.506501 26210 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.509153 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.509863 23890 catalog_manager.cc:5484] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 3, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:37.512038 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.517297 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.520973 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.521687 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.523264 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.523402 24429 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.523492 24429 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.528153 24429 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.528875 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.529004 24298 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.530014 24298 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.530318 24366 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.530468 24429 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.530712 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.530848 24035 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.531886 24035 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.533411 26346 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.533524 26346 raft_consensus.cc:686] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.533687 26346 consensus_queue.cc:237] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.537078 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.537195 24430 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.537256 24430 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.538038 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.538229 24430 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.538552 24430 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.539093 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.539216 24035 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.539258 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15ff4003c1654707a17d65993fb938af"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.539364 24430 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.539430 24430 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.539865 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.539988 24165 raft_consensus.cc:3032] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.540045 24165 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.540195 24165 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.540279 24035 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.540274 24430 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.540351 24165 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.540925 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.541050 24298 raft_consensus.cc:3032] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.541105 24298 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.541257 24298 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.541384 24165 raft_consensus.cc:2396] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
I20240501 07:59:37.541458 24298 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.542400 24298 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.542346 23890 catalog_manager.cc:5484] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 2 to 4, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:37.542883 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.544540 24366 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.544726 26346 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.544818 26346 raft_consensus.cc:686] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.544958 26346 consensus_queue.cc:237] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.548158 24430 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.548775 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.548874 24363 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.548897 24035 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.549561 24298 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240501 07:59:37.549887 24035 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.550629 25392 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.551105 25392 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.551187 25392 raft_consensus.cc:686] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.551294 25392 consensus_queue.cc:237] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.553575 24036 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240501 07:59:37.554983 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.555138 24430 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.555209 24430 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.555466 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.557561 24430 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.557845 24430 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.559309 26221 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:37.560846 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.560950 24167 raft_consensus.cc:3032] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.560992 24167 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.560848 23890 catalog_manager.cc:5484] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.561110 24167 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.561146 23890 catalog_manager.cc:5484] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:37.561245 24167 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.562301 24167 raft_consensus.cc:2396] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240501 07:59:37.562448 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.562569 24296 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.563711 24296 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.565508 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.566066 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.566076 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.567705 24165 raft_consensus.cc:1261] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.568466 24366 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters: 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.568902 24430 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.569008 24430 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.570109 24430 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.570844 25392 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.570948 25392 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.571103 25392 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.572170 26221 consensus_queue.cc:1035] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.572266 26221 consensus_queue.cc:1225] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer 3748d2c0aeb740f98f8c2caabad21474 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.573765 24165 pending_rounds.cc:76] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 3
I20240501 07:59:37.573820 24165 pending_rounds.cc:97] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 07:59:37.573880 24165 raft_consensus.cc:2866] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.575511 24296 raft_consensus.cc:1261] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:37.575851 24430 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.576799 25392 consensus_queue.cc:1035] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.577899 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.582363 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.582481 24036 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.583041 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.583154 24165 raft_consensus.cc:3032] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.583194 24165 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.583317 24165 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.583417 24165 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.584815 24165 raft_consensus.cc:2396] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 15.
W20240501 07:59:37.585455 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.590036 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.591238 24036 raft_consensus.cc:2396] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 15.
I20240501 07:59:37.591346 24035 raft_consensus.cc:1226] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 3. Current term is 4. Ops: []
W20240501 07:59:37.591734 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.592193 24429 raft_consensus.cc:1226] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 3. Current term is 4. Ops: []
W20240501 07:59:37.593179 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.593843 23895 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:37.594501 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.595404 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.599061 24036 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240501 07:59:37.600044 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.600147 24429 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.600219 24429 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.601186 24429 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.601496 24429 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.601876 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.601989 24296 raft_consensus.cc:3032] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.602032 24296 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:37.602161 24296 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.602330 24296 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.603268 24296 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.603545 24366 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.603720 26221 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.603811 26221 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.603940 26221 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.604331 24167 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240501 07:59:37.606052 26221 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:59:37.606149 26221 consensus_queue.cc:1225] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer 3748d2c0aeb740f98f8c2caabad21474 log is divergent from this leader: its last log entry 3.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240501 07:59:37.608633 24363 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.609148 26221 raft_consensus.cc:2726] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240501 07:59:37.609519 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.609760 26221 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:59:37.613289 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57390: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.613601 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.613749 24036 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.614241 24165 pending_rounds.cc:76] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 11
I20240501 07:59:37.614291 24165 pending_rounds.cc:97] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 3.12
I20240501 07:59:37.614320 24165 raft_consensus.cc:2866] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 07:59:37.615243 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.616250 23895 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.617642 26015 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:37.617908 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.618029 24430 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.618081 24430 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.619239 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.619465 24430 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.620092 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.620210 24296 raft_consensus.cc:3032] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.620252 24296 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
W20240501 07:59:37.620261 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.620390 24296 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.620568 24296 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.621568 24296 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.621958 24366 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.622972 25392 raft_consensus.cc:2781] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.623098 25392 raft_consensus.cc:686] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.623236 25392 consensus_queue.cc:237] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.624084 24036 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
I20240501 07:59:37.624375 24296 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
I20240501 07:59:37.624764 26347 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20240501 07:59:37.625519 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.628120 26201 mvcc.cc:205] Tried to move back new op lower bound from 7022798346723385344 to 7022798346667290624. Current Snapshot: MvccSnapshot[applied={T|T < 7022798346723385344}]
W20240501 07:59:37.632046 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.633158 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.636535 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.637558 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.639590 24036 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
I20240501 07:59:37.640161 26221 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 07:59:37.643689 24430 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
W20240501 07:59:37.643811 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.651523 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.653873 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.655148 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.655277 24035 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.655483 23895 catalog_manager.cc:5484] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.656385 24035 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.657302 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.663640 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.664435 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0272069f81436f81725c3092e0e382"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.664548 24429 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.664605 24429 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.665686 24429 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.666023 24429 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.667209 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.667331 24165 raft_consensus.cc:3032] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.667377 24165 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.667529 24165 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.667693 24165 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.668655 24165 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.669835 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.676620 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.677768 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.677829 24296 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.677892 24034 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.678233 26341 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 07:59:37.678752 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.679036 24034 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 4.
W20240501 07:59:37.679540 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.682991 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.684286 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.685158 24034 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.685467 25335 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 07:59:37.686230 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.688813 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.691777 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.693262 24363 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.693428 26347 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.693840 26347 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.694010 26347 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.696578 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.696561 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:37.696679 24429 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.697113 24429 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.697420 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.698163 24429 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.698469 24429 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:37.698833 24429 raft_consensus.cc:1226] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 3. Current term is 4. Ops: [3.14-3.14]
I20240501 07:59:37.700109 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.700279 24165 raft_consensus.cc:3032] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.700325 24165 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.700454 24165 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.700719 24165 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.700990 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.701731 24165 raft_consensus.cc:2396] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13.
I20240501 07:59:37.702549 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:37.702702 24036 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.703631 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.704157 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.704557 23895 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 4 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:37.705132 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.706450 24036 raft_consensus.cc:2396] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 4 because replica has last-logged OpId of term: 3 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13.
W20240501 07:59:37.712036 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.713076 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.713950 24363 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474
I20240501 07:59:37.714478 26221 raft_consensus.cc:2726] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240501 07:59:37.716964 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.721468 24165 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
I20240501 07:59:37.721750 24167 raft_consensus.cc:1261] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.722059 24298 raft_consensus.cc:1261] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 07:59:37.722470 26347 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.722649 24036 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 4 index: 22. (index mismatch)
I20240501 07:59:37.722816 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.722956 24035 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.723042 24035 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.723085 25392 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 07:59:37.723347 25392 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.723429 25392 consensus_queue.cc:1225] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer 3748d2c0aeb740f98f8c2caabad21474 log is divergent from this leader: its last log entry 2.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240501 07:59:37.724162 24035 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.724524 25392 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 07:59:37.724567 24035 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.724889 24164 pending_rounds.cc:76] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 7
I20240501 07:59:37.724943 24164 pending_rounds.cc:97] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 2.8
I20240501 07:59:37.724972 24164 raft_consensus.cc:2866] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.728333 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.728464 24296 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.729575 24296 raft_consensus.cc:2396] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 07:59:37.730494 24034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.730705 24034 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.730805 24034 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.731477 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.732065 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.732182 24429 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.732458 24034 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.733251 24429 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.733783 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.733901 24165 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.735863 23971 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.736135 26213 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.736233 26213 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.734968 24165 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.736377 26213 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.736557 23970 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:37.736711 26271 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.736856 26271 raft_consensus.cc:686] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.736989 26271 consensus_queue.cc:237] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.739123 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.740129 24034 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.740150 23895 catalog_manager.cc:5484] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.740397 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.740406 23895 catalog_manager.cc:5484] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.740506 24297 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.741634 24297 raft_consensus.cc:2396] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20240501 07:59:37.744925 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.747459 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.747808 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.747925 24036 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.747979 24036 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.748308 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.748977 24036 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.749351 24036 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.749567 24037 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 07:59:37.750278 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.750380 24298 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.751488 24298 raft_consensus.cc:2396] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 07:59:37.751818 24165 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.752166 26271 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.752301 24165 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 07:59:37.753145 26221 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:59:37.753284 26221 consensus_queue.cc:1225] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer 3748d2c0aeb740f98f8c2caabad21474 log is divergent from this leader: its last log entry 2.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20240501 07:59:37.753898 24166 pending_rounds.cc:76] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 13
I20240501 07:59:37.753957 24166 pending_rounds.cc:97] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 2.14
I20240501 07:59:37.753991 24166 raft_consensus.cc:2866] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.756011 26177 mvcc.cc:205] Tried to move back new op lower bound from 7022798347267268608 to 7022798347211960320. Current Snapshot: MvccSnapshot[applied={T|T < 7022798347267268608}]
I20240501 07:59:37.757301 24297 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 07:59:37.758087 26346 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 07:59:37.758186 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.758939 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.760267 24167 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 07:59:37.760669 25392 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.760854 24034 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 07:59:37.761107 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.761205 24035 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.761255 24035 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.761593 25392 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:37.762311 24035 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.762671 24035 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.763010 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.763134 24297 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.763317 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.763414 24429 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.763427 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.763990 24297 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.764382 24429 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.764794 26346 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:37.764902 26346 consensus_queue.cc:1225] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer 3748d2c0aeb740f98f8c2caabad21474 log is divergent from this leader: its last log entry 2.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240501 07:59:37.765235 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.765358 24167 raft_consensus.cc:3032] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.765398 24167 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.765515 24167 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.765671 24167 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.765923 23971 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.766667 24164 pending_rounds.cc:76] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 9
I20240501 07:59:37.766721 24164 pending_rounds.cc:97] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 2.10
I20240501 07:59:37.766764 24164 raft_consensus.cc:2866] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.767714 26271 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.767802 26271 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.767963 26271 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:37.769608 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.769618 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.769977 24297 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 8. (term mismatch)
I20240501 07:59:37.770033 24297 pending_rounds.cc:76] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 7
I20240501 07:59:37.770059 24297 pending_rounds.cc:97] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted WRITE_OP operation due to leader change: 2.8
W20240501 07:59:37.770172 24297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57374: Aborted: Op aborted by new leader
I20240501 07:59:37.770617 23973 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.766673 24167 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.771250 26213 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.771322 26213 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.771447 26213 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.772130 26221 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.773368 24037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.773471 24037 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.773521 24037 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.773784 24296 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.774541 24037 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.774874 24037 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
W20240501 07:59:37.774978 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.775471 24033 raft_consensus.cc:1261] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
W20240501 07:59:37.776329 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.776367 23895 catalog_manager.cc:5484] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.779166 24035 raft_consensus.cc:1226] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 3. Current term is 4. Ops: []
I20240501 07:59:37.779474 24429 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.779870 25335 consensus_queue.cc:1035] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:59:37.780162 26271 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.780254 26271 consensus_queue.cc:1225] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.780579 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.780736 24296 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.780874 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.781114 26271 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.781791 24296 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.782583 25392 consensus_queue.cc:1046] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: INVALID_TERM, Last received: 3.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.782804 25392 raft_consensus.cc:3032] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.782848 25392 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.783748 25392 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.783923 25392 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:37.784874 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.785961 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53300: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.786041 23973 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.786443 24299 pending_rounds.cc:76] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.786499 24299 pending_rounds.cc:97] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 07:59:37.786532 24299 raft_consensus.cc:2866] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.787111 26271 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.787199 26271 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.787324 26271 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.788166 24035 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.788286 24035 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.788340 24035 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.789093 24296 raft_consensus.cc:1261] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 13. (term mismatch)
I20240501 07:59:37.789150 24296 pending_rounds.cc:76] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 12
I20240501 07:59:37.789175 24296 pending_rounds.cc:97] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 2.13
I20240501 07:59:37.789202 24296 raft_consensus.cc:2866] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.789387 24035 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.789721 24035 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.789718 26341 consensus_queue.cc:1035] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:59:37.790158 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.790900 26147 heartbeater.cc:503] Master 127.0.99.190:46851 was elected leader, sending a full tablet report...
I20240501 07:59:37.790913 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.790995 24298 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.791051 24167 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.792084 24167 raft_consensus.cc:2396] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
I20240501 07:59:37.792153 24429 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.792749 26271 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.792860 26271 consensus_queue.cc:1225] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.793310 24297 raft_consensus.cc:1226] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 3. Current term is 4. Ops: [3.8-3.8]
W20240501 07:59:37.795280 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.795336 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.796213 23895 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 4 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:37.799103 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.799589 24299 pending_rounds.cc:76] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.799643 24299 pending_rounds.cc:97] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 07:59:37.799674 24299 raft_consensus.cc:2866] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.799763 24167 raft_consensus.cc:1226] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 4. Current term is 5. Ops: []
I20240501 07:59:37.800469 26243 mvcc.cc:205] Tried to move back new op lower bound from 7022798347267268608 to 7022798343549919232. Current Snapshot: MvccSnapshot[applied={T|T < 7022798347267268608}]
I20240501 07:59:37.801347 26355 mvcc.cc:205] Tried to move back new op lower bound from 7022798347352641536 to 7022798343541444608. Current Snapshot: MvccSnapshot[applied={T|T < 7022798347352641536}]
I20240501 07:59:37.802289 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.802731 24036 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.802798 24036 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.803961 24036 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.804662 24036 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.804922 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.805037 24164 raft_consensus.cc:3032] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.805079 24164 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.805223 24164 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.805369 24164 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.805956 26347 consensus_queue.cc:1046] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: INVALID_TERM, Last received: 4.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 07:59:37.806244 26347 raft_consensus.cc:3032] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.806295 26347 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.806416 26347 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.18, Last appended by leader: 18, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.806495 24164 raft_consensus.cc:2396] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 4 because replica has last-logged OpId of term: 3 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 9.
I20240501 07:59:37.806555 26347 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.808063 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.808310 23890 catalog_manager.cc:5484] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 4, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 4 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:37.814412 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.814450 24428 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240501 07:59:37.814910 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.815053 24298 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.815778 24033 raft_consensus.cc:1226] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 3. Current term is 4. Ops: [3.10-3.10]
I20240501 07:59:37.816154 24298 raft_consensus.cc:2396] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 4 because replica has last-logged OpId of term: 3 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 9.
W20240501 07:59:37.816253 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.816437 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.817668 26373 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:37.818423 24297 raft_consensus.cc:1261] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
W20240501 07:59:37.818936 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.819236 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.819422 24429 raft_consensus.cc:2396] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
I20240501 07:59:37.819718 23973 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.819898 26271 raft_consensus.cc:2726] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.820067 26271 consensus_queue.cc:1035] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:37.821225 24036 raft_consensus.cc:1226] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 4. Current term is 5. Ops: [4.18-4.18]
I20240501 07:59:37.821645 24033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.821754 24033 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.821806 24033 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.822657 24033 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.823101 24033 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.823715 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.823828 24164 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.824877 24164 raft_consensus.cc:2396] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
W20240501 07:59:37.825778 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.826711 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.830303 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.830433 24430 raft_consensus.cc:3032] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.830480 24430 raft_consensus.cc:729] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.830601 24430 consensus_queue.cc:260] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.830742 24430 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.831578 24430 raft_consensus.cc:2396] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
W20240501 07:59:37.831895 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.837034 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.839327 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.838979 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.840106 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50008: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.841044 23971 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.841343 23971 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.841406 26370 raft_consensus.cc:2726] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 07:59:37.841852 26213 raft_consensus.cc:2726] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240501 07:59:37.842978 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.844048 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57374: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.846494 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.846493 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.846596 24036 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.846648 24036 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.847713 24036 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.848073 24036 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.848486 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.848593 24167 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.849596 24167 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.849853 23970 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:37.850041 26213 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.850142 26213 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
W20240501 07:59:37.850281 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.850258 26213 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.851169 24298 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 4 index: 12. (index mismatch)
I20240501 07:59:37.851563 25392 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:37.852036 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.852141 24296 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.852592 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.852703 24036 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.852756 24036 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.853207 24296 raft_consensus.cc:2396] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 07:59:37.853663 24036 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.853796 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.853960 24036 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.854468 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.854562 24164 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.854527 23895 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.855916 24164 raft_consensus.cc:2396] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 4 because replica has last-logged OpId of term: 3 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 14.
W20240501 07:59:37.856549 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.857239 24164 raft_consensus.cc:1261] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 4 index: 12. (index mismatch)
I20240501 07:59:37.858160 26347 consensus_queue.cc:1035] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 07:59:37.858239 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.860191 24164 raft_consensus.cc:1226] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 3. Current term is 4. Ops: []
W20240501 07:59:37.864162 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.864570 26369 consensus_queue.cc:1046] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: INVALID_TERM, Last received: 3.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:37.868891 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.871340 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.871477 24036 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.871537 24036 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.872807 24033 raft_consensus.cc:1226] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 3. Current term is 4. Ops: [3.15-3.15]
I20240501 07:59:37.873013 24036 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.873368 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50008: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.873395 24036 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.874655 24430 raft_consensus.cc:3032] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.874738 24430 raft_consensus.cc:729] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.874876 24430 consensus_queue.cc:260] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.874955 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.874998 24430 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.875444 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.875576 24296 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.876053 24430 raft_consensus.cc:2396] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 4 because replica has last-logged OpId of term: 3 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 14.
I20240501 07:59:37.876523 24296 raft_consensus.cc:2396] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 10.
W20240501 07:59:37.876199 26346 consensus_queue.cc:1159] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" responder_term: 4 status { last_received { term: 3 index: 14 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3b8f81da8ae74d3687ef72a2c9ce0888 for earlier term 3. Current term is 4. Ops: [3.15-3.15]" } } last_received_current_leader { term: 0 index: 0 } }
W20240501 07:59:37.877439 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.877779 25392 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240501 07:59:37.878082 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.878171 24430 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.877614 23971 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.879047 24033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15ff4003c1654707a17d65993fb938af"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.879083 24430 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.879158 24033 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.879179 26271 raft_consensus.cc:2726] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 07:59:37.879209 24033 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.879423 23971 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:37.880074 24033 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:37.880277 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.880427 26213 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.880508 26213 raft_consensus.cc:686] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
W20240501 07:59:37.880517 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.880625 26213 consensus_queue.cc:237] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.10, Last appended by leader: 10, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:37.880842 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57374: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.880434 24033 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
W20240501 07:59:37.881928 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.882123 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.882191 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.882226 24430 raft_consensus.cc:3032] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.882262 24430 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.882285 24298 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.882377 24430 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.9, Last appended by leader: 9, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.882503 24430 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.882846 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.883390 24298 raft_consensus.cc:2454] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.883450 24430 raft_consensus.cc:2396] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240501 07:59:37.883883 23973 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.883824 23893 catalog_manager.cc:5484] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.884035 26213 raft_consensus.cc:2781] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.884106 26213 raft_consensus.cc:686] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.884215 26213 consensus_queue.cc:237] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:37.885047 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.885147 24036 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.885205 24036 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.885847 24164 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
W20240501 07:59:37.886087 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.886101 24036 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.886160 26271 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:37.886372 24036 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.886812 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.886837 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.886904 24299 raft_consensus.cc:1261] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240501 07:59:37.886919 24167 raft_consensus.cc:3032] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:37.886926 24296 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.886960 24167 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.887099 24167 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.887274 24167 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:37.887801 24296 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.888195 24167 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:37.889146 23970 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
W20240501 07:59:37.889246 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.889282 26213 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:37.889361 26213 raft_consensus.cc:686] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.889945 26213 consensus_queue.cc:237] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.890583 23890 catalog_manager.cc:5484] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:37.891218 26373 consensus_queue.cc:1035] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:37.893450 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.895500 23893 catalog_manager.cc:5484] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 4, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 4 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:37.897444 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.898283 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.898386 24036 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.898440 24036 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.899296 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.899420 24036 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.900048 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.900148 24164 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.900517 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.901119 24164 raft_consensus.cc:2396] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
I20240501 07:59:37.901662 24036 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
W20240501 07:59:37.903182 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.903775 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.903882 24428 raft_consensus.cc:3032] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.903918 24428 raft_consensus.cc:729] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.904044 24428 consensus_queue.cc:260] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.10, Last appended by leader: 10, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.904183 24428 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.904508 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.905241 24428 raft_consensus.cc:2396] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 9.
W20240501 07:59:37.905797 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57394: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.906121 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.907060 23971 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f; no voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.907354 26373 raft_consensus.cc:2726] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240501 07:59:37.908993 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.910595 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.910708 24036 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.910785 24036 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.911936 24036 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.912261 24036 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:37.912657 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.912771 24428 raft_consensus.cc:3032] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.912822 24428 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
W20240501 07:59:37.912863 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.912937 24428 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 15, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.913089 24428 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.913375 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.913476 24298 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.914014 24428 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.914453 24298 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.914741 24167 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 7. (term mismatch)
I20240501 07:59:37.914794 24167 pending_rounds.cc:76] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 6
I20240501 07:59:37.914819 24167 pending_rounds.cc:97] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 3.7
I20240501 07:59:37.914845 24167 raft_consensus.cc:2866] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.915633 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.915735 24036 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.915786 24036 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.916213 26369 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:37.916640 24036 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.917171 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.917271 24167 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.917311 24167 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:37.917790 23973 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:37.918395 26271 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.918498 26271 raft_consensus.cc:686] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.918609 26271 consensus_queue.cc:237] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 15, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.918766 24296 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.919165 24428 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20240501 07:59:37.919917 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.920207 24167 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 4.14, Last appended by leader: 14, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.920384 24167 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.921654 24167 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.921952 23970 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:37.922143 26370 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.922171 24296 raft_consensus.cc:1261] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240501 07:59:37.922216 26370 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.922329 26370 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 4.14, Last appended by leader: 14, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.922381 26373 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.922444 26373 consensus_queue.cc:1225] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.922528 26221 consensus_queue.cc:1035] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:37.923614 24389 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.924700 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.925348 24299 pending_rounds.cc:76] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.925407 24299 pending_rounds.cc:97] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 07:59:37.925444 24299 raft_consensus.cc:2866] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:37.926460 26370 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.928968 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.930001 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.930794 24166 raft_consensus.cc:1261] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.931228 26373 consensus_queue.cc:1035] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:37.931963 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.933777 23895 catalog_manager.cc:5484] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.936246 24299 raft_consensus.cc:1261] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 07:59:37.936973 26370 consensus_queue.cc:1035] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:37.937067 26370 consensus_queue.cc:1225] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 07:59:37.938124 24298 pending_rounds.cc:76] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 3
I20240501 07:59:37.938177 24298 pending_rounds.cc:97] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 07:59:37.938207 24298 raft_consensus.cc:2866] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 07:59:37.941462 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.941989 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.945519 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.948472 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.948952 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50074: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.950312 24036 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.951640 23893 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.953307 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.953428 24299 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.954638 24299 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.957691 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.957819 24036 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.957880 24036 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.958897 24036 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.959262 24036 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.960206 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.960320 24167 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.960650 24298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.960767 24298 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.961490 24167 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.961752 23970 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:37.961849 24298 raft_consensus.cc:2396] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240501 07:59:37.961933 26373 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.962011 26373 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.962167 26373 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:37.964032 23895 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:37.965683 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57222: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:37.966075 24128 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.966694 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.967135 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.967698 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.967818 24036 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.967882 24036 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.968910 24036 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.969272 24036 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
W20240501 07:59:37.969798 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:37.975191 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.975324 24428 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.976297 24428 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.976706 23971 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.976905 26373 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.976984 26373 raft_consensus.cc:686] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.977079 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:37.977114 26373 consensus_queue.cc:237] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:37.977166 24036 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.977211 24036 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.978134 24036 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.978458 24036 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:37.978996 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:37.979127 24296 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.979163 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.979276 24428 raft_consensus.cc:3032] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.979314 24428 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:37.979458 24428 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.979574 24428 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.980094 24296 raft_consensus.cc:2454] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.980717 24428 raft_consensus.cc:2396] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1df5867ab7af4bd0a8229838d80e9d7f for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240501 07:59:37.981858 24146 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:59:37.982158 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.982775 23973 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters: 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:37.982913 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0272069f81436f81725c3092e0e382"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
W20240501 07:59:37.983749 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:37.985073 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.985369 24036 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.985455 24036 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:37.985950 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.986415 23895 catalog_manager.cc:5484] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:37.986794 24036 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.987152 24036 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:37.989789 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:37.989903 24428 raft_consensus.cc:3032] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:37.989941 24428 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
W20240501 07:59:37.989987 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.990082 24428 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.990252 24428 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.992861 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:37.993007 24164 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:37.993449 25949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:59:37.993947 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53286: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:37.993988 24428 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:37.994261 23971 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:37.994421 26373 raft_consensus.cc:2781] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:37.994493 26373 raft_consensus.cc:686] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:37.994608 26373 consensus_queue.cc:237] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:37.995992 24298 raft_consensus.cc:1261] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240501 07:59:37.996466 24166 raft_consensus.cc:1261] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240501 07:59:37.997133 26373 consensus_queue.cc:1035] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:37.997226 26373 consensus_queue.cc:1225] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 07:59:37.997035 23895 catalog_manager.cc:5484] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:37.997539 24409 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:37.998556 26271 consensus_queue.cc:1035] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:37.998826 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.002904 24015 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:38.003088 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
W20240501 07:59:38.003134 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.003314 24167 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.006065 24167 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.019469 24164 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.021407 24299 pending_rounds.cc:76] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Aborting all ops after (but not including) 4
I20240501 07:59:38.021468 24299 pending_rounds.cc:97] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 07:59:38.021497 24299 raft_consensus.cc:2866] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:38.022872 26213 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.022966 26213 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.023119 26213 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:38.023259 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.024546 23995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57240: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.024602 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.025595 24299 raft_consensus.cc:1261] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 07:59:38.025929 26369 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:38.028148 24167 raft_consensus.cc:1261] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20240501 07:59:38.028213 24167 pending_rounds.cc:76] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Aborting all ops after (but not including) 5
I20240501 07:59:38.028239 24167 pending_rounds.cc:97] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240501 07:59:38.028265 24167 raft_consensus.cc:2866] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:38.028623 26221 consensus_queue.cc:1035] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:38.028585 23895 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.028999 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:38.029106 24036 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.029155 24036 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.030148 24036 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.030256 24164 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 5 index: 16. (index mismatch)
I20240501 07:59:38.034693 26373 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:38.034922 24036 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.037698 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:38.038163 24036 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.038233 24036 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.039307 24036 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.039624 24036 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.040998 26081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:59:38.042472 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.050956 25818 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:38.052664 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.052794 24165 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.053848 24165 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.054975 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.055193 24166 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.056914 24166 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
I20240501 07:59:38.059667 24279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:59:38.068192 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.070676 23970 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.070839 26373 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.070916 26373 raft_consensus.cc:686] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.071416 26373 consensus_queue.cc:237] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:38.071714 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.072036 23970 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
W20240501 07:59:38.073558 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.075549 24428 raft_consensus.cc:1261] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 5 index: 16. (index mismatch)
I20240501 07:59:38.075765 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.075861 24429 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.076892 24429 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.077446 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.077544 24299 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.078691 24299 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 4.
W20240501 07:59:38.082641 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.086387 25687 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:38.087719 26398 raft_consensus.cc:2781] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:38.087838 26398 raft_consensus.cc:686] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.087980 26398 consensus_queue.cc:237] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:38.088783 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.089265 23893 catalog_manager.cc:5484] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 3 to 5, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:38.094681 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.096534 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.099489 23893 catalog_manager.cc:5484] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 2 to 4, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 4 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.102952 24299 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20240501 07:59:38.104497 24430 raft_consensus.cc:1261] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
W20240501 07:59:38.105165 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.106447 24036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea"
dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
from {username='slave'} at 127.0.0.1:57184
I20240501 07:59:38.106567 24036 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.106637 24036 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.107645 24036 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.107995 24036 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
W20240501 07:59:38.109380 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50060: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.110159 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.110924 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.111775 26213 consensus_queue.cc:1035] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:59:38.112298 26373 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:38.112402 26373 consensus_queue.cc:1225] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 3b8f81da8ae74d3687ef72a2c9ce0888 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240501 07:59:38.112581 26398 consensus_queue.cc:1035] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:38.114109 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 25 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.114233 24430 raft_consensus.cc:3032] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.114280 24430 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.114426 24430 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 4.25, Last appended by leader: 25, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:38.114502 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.114637 24430 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.115675 24430 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.124008 24166 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20240501 07:59:38.124477 26373 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 07:59:38.124641 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" candidate_term: 5 candidate_status { last_received { term: 4 index: 25 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.124799 24299 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.125639 24299 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1df5867ab7af4bd0a8229838d80e9d7f in term 5.
I20240501 07:59:38.130936 26355 mvcc.cc:205] Tried to move back new op lower bound from 7022798348742373376 to 7022798348582027264. Current Snapshot: MvccSnapshot[applied={T|T < 7022798348742373376 or (T in {7022798348742373376})}]
I20240501 07:59:38.132212 24430 raft_consensus.cc:1261] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20240501 07:59:38.133591 24298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.133699 24298 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.133745 24298 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.134747 24298 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.135185 24298 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.135671 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.135783 24165 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.135811 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.135942 24036 raft_consensus.cc:3032] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.135989 24036 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.136121 24036 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.11, Last appended by leader: 11, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.136274 24036 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.136752 24165 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.137055 24036 raft_consensus.cc:2454] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.137087 24232 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.137241 25421 raft_consensus.cc:2781] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.137310 25421 raft_consensus.cc:686] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.137429 25421 consensus_queue.cc:237] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.11, Last appended by leader: 11, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.137682 24298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
W20240501 07:59:38.137709 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.137790 24298 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.137837 24298 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.138659 24298 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.139042 24298 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.139290 23895 catalog_manager.cc:5484] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.139362 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.139467 24429 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.140502 24429 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
W20240501 07:59:38.140983 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.141062 23973 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.142071 26213 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.142153 26213 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Leader. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.142263 26213 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 4.25, Last appended by leader: 25, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.143329 24428 pending_rounds.cc:76] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Aborting all ops after (but not including) 6
I20240501 07:59:38.143383 24428 pending_rounds.cc:97] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20240501 07:59:38.143450 24428 raft_consensus.cc:2866] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 07:59:38.143568 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.146164 24233 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.147248 23895 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.148430 25421 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.148532 25421 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.148662 25421 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.150043 24165 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240501 07:59:38.151285 26273 mvcc.cc:205] Tried to move back new op lower bound from 7022798348658884608 to 7022798348267302912. Current Snapshot: MvccSnapshot[applied={T|T < 7022798348658884608}]
I20240501 07:59:38.151417 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.151527 24036 raft_consensus.cc:3032] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.151569 24036 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.151706 24036 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.151862 24036 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.152233 24429 raft_consensus.cc:1226] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: []
I20240501 07:59:38.152998 24036 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.153842 26408 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 07:59:38.155217 23893 catalog_manager.cc:5484] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.156090 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.156211 24296 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.156275 24296 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.157238 24296 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.157655 24296 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.159322 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
W20240501 07:59:38.159363 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.159437 24429 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.159830 26213 consensus_queue.cc:1035] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:38.160471 24429 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.160763 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.160892 24036 raft_consensus.cc:3032] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.160943 24036 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.161080 24036 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.161758 24233 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.161949 25421 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.162025 25421 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.162143 25421 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.162372 24429 raft_consensus.cc:1261] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 5 index: 24. (index mismatch)
I20240501 07:59:38.162679 24430 raft_consensus.cc:1226] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: []
W20240501 07:59:38.163780 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.164366 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.164495 24296 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.164551 24296 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.165572 24296 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.165964 24296 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.165990 23895 catalog_manager.cc:5484] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.166347 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.166440 24428 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.166733 26201 mvcc.cc:205] Tried to move back new op lower bound from 7022798348658884608 to 7022798348267302912. Current Snapshot: MvccSnapshot[applied={T|T < 7022798348658884608}]
I20240501 07:59:38.167192 24033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.167306 24033 raft_consensus.cc:3032] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.167366 24033 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.167469 24428 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.167517 24033 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.14, Last appended by leader: 14, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.167543 26398 consensus_queue.cc:1035] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 07:59:38.169149 24233 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.169328 26210 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.169411 26210 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.169533 26210 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.14, Last appended by leader: 14, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:38.170733 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.172959 23890 catalog_manager.cc:5484] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.173377 24036 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.174418 24036 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.174810 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.174924 24296 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.174986 24296 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.175935 24296 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.176525 24296 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.176959 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.177080 24165 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.177101 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.177210 24428 raft_consensus.cc:3032] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:38.177245 24428 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
W20240501 07:59:38.177350 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.177390 24428 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.177568 24428 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.178516 24428 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.178535 24165 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.178835 24233 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.179080 25421 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:38.179170 25421 raft_consensus.cc:686] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.179303 25421 consensus_queue.cc:237] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.180922 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.180917 23893 catalog_manager.cc:5484] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 3 to 4, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.181030 24299 raft_consensus.cc:482] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.181080 24299 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.181466 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.182354 24299 raft_consensus.cc:504] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.182849 24299 leader_election.cc:288] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.183169 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.183271 24036 raft_consensus.cc:3032] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.183310 24036 raft_consensus.cc:729] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.183420 24036 consensus_queue.cc:260] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.9, Last appended by leader: 9, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.183518 24036 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.183617 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609f9ef5e1fa42daabc3c8fb9fb00613" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.183738 24428 raft_consensus.cc:3037] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.183848 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53272: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.184355 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.184453 24296 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.184478 24036 raft_consensus.cc:2396] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20240501 07:59:38.184509 24296 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.184649 24428 raft_consensus.cc:2396] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20240501 07:59:38.185042 24427 raft_consensus.cc:1226] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: []
I20240501 07:59:38.185034 24233 leader_election.cc:302] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d371a71c02a4432aca2fed33b2545d5; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:38.185197 26210 raft_consensus.cc:2726] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 07:59:38.186272 24296 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.186599 24296 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.186944 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.187078 24167 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.187062 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.187158 24036 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.187199 24430 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 25. Preceding OpId from leader: term: 5 index: 27. (index mismatch)
I20240501 07:59:38.187646 26370 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 07:59:38.188004 24167 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.188060 24036 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.188544 24232 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.188797 26210 raft_consensus.cc:2781] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.188874 26210 raft_consensus.cc:686] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.188999 26210 consensus_queue.cc:237] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.189165 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.189293 24296 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.189356 24296 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.190147 24296 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.190613 24296 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.191282 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.191396 24296 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.191457 24296 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.191596 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.191717 24167 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.192623 24296 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.192668 24167 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.193254 24232 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.193418 25421 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.193495 25421 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
W20240501 07:59:38.193519 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.193647 25421 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.18, Last appended by leader: 18, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.193608 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.193758 24430 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.193794 24430 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.193941 24430 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.18, Last appended by leader: 18, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.194383 24430 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:38.194551 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.194785 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.194885 24427 raft_consensus.cc:3032] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:38.194955 24427 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.195097 24427 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.195262 24427 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.195380 23895 catalog_manager.cc:5484] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 3 to 5, leader changed from 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.195510 24430 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.196220 24427 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.196329 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.196419 24167 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.196602 24296 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.196619 24233 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.197151 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.197230 24296 raft_consensus.cc:482] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.197265 24296 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.197239 24167 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.197527 26210 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:38.197618 26210 raft_consensus.cc:686] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.197743 26210 consensus_queue.cc:237] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.197939 24296 raft_consensus.cc:504] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.198626 24296 leader_election.cc:288] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.199254 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.199357 24427 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.200222 24427 raft_consensus.cc:2454] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.200240 23895 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.200460 23895 catalog_manager.cc:5484] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 3 to 4, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.200498 24233 leader_election.cc:302] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
W20240501 07:59:38.201207 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.202680 25421 raft_consensus.cc:2781] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.202764 25421 raft_consensus.cc:686] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.202906 25421 consensus_queue.cc:237] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 15, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:38.202633 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.199154 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090ee3c1fc884691810492f5eccbef67" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 4 index: 15 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.204418 24036 raft_consensus.cc:3032] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.204459 24036 raft_consensus.cc:729] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.204571 24036 consensus_queue.cc:260] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 5.16, Last appended by leader: 16, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.204707 24036 raft_consensus.cc:3037] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.205790 24036 raft_consensus.cc:2396] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 15.
I20240501 07:59:38.205595 24298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.206408 24298 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.206485 24298 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.207731 24298 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.207763 23895 catalog_manager.cc:5484] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:38.208078 23973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.131:43575, user_credentials={real_user=slave}} blocked reactor thread for 46136.5us
I20240501 07:59:38.208204 24033 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.209141 24033 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.209321 24296 raft_consensus.cc:1226] T 609f9ef5e1fa42daabc3c8fb9fb00613 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: [5.9-5.9]
W20240501 07:59:38.209534 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.211392 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.214190 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.214303 24430 raft_consensus.cc:3032] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.214341 24430 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.214465 24430 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.214643 24430 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.215703 24430 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.216233 24233 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.216499 26210 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.216600 26210 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.216729 26210 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
W20240501 07:59:38.217677 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50032: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.218644 24427 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20240501 07:59:38.218988 25421 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 07:59:38.219614 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.220155 24258 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57358: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.220513 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.223819 24033 raft_consensus.cc:1261] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20240501 07:59:38.223876 24165 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
I20240501 07:59:38.225437 24296 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 14. Preceding OpId from leader: term: 5 index: 16. (index mismatch)
I20240501 07:59:38.225780 26373 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:38.225951 24167 raft_consensus.cc:1261] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 14. Preceding OpId from leader: term: 5 index: 16. (index mismatch)
W20240501 07:59:38.227447 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.226773 26213 consensus_queue.cc:1035] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20240501 07:59:38.230214 24033 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
W20240501 07:59:38.233345 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.234161 23997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.235323 26210 consensus_queue.cc:1035] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:38.235361 26438 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:38.238873 26210 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:59:38.240918 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50048: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.243168 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.243937 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.244237 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.247833 23895 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:38.252094 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.254604 24298 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.255193 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.255317 24297 raft_consensus.cc:482] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.255385 24297 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.256582 24297 raft_consensus.cc:504] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.256989 24297 leader_election.cc:288] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.257416 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.257514 24428 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.258415 24428 raft_consensus.cc:2454] T e63f2991bfdf4e4a9aec4534605cf82e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.259294 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.259373 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15ff4003c1654707a17d65993fb938af"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.259402 24167 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.259467 24297 raft_consensus.cc:482] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.259516 24297 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.259675 24233 leader_election.cc:302] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.259846 26210 raft_consensus.cc:2781] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.259940 26210 raft_consensus.cc:686] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.260047 26210 consensus_queue.cc:237] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.260391 24297 raft_consensus.cc:504] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.260677 24297 leader_election.cc:288] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.261814 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.261919 24428 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.262782 24428 raft_consensus.cc:2396] T 15ff4003c1654707a17d65993fb938af P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 4 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240501 07:59:38.262966 24164 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 11. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20240501 07:59:38.263707 26210 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:59:38.263854 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.264611 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.264711 24297 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.264760 24297 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:38.265142 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.265594 24297 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.265959 24297 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.266476 24295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.266582 24295 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.266633 24295 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.267092 24167 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
W20240501 07:59:38.267488 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.267585 24295 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.267881 24295 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.268898 24033 raft_consensus.cc:1261] T 6523f8bf6c574f36bd31be66abf6fd23 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 11. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20240501 07:59:38.269266 24033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63f2991bfdf4e4a9aec4534605cf82e" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.269366 24033 raft_consensus.cc:3032] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.269404 24033 raft_consensus.cc:729] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.269373 24295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.269465 24295 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.269517 24295 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.269526 24033 consensus_queue.cc:260] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.11, Last appended by leader: 11, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
W20240501 07:59:38.268898 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.269667 24033 raft_consensus.cc:3037] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.270234 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.270331 24168 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.270488 24295 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.270617 24033 raft_consensus.cc:2396] T e63f2991bfdf4e4a9aec4534605cf82e P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240501 07:59:38.270799 24295 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.271378 24168 raft_consensus.cc:2454] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.271404 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.271481 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.271541 24164 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.271572 24163 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.272164 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.272449 24164 raft_consensus.cc:2454] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.272553 24232 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.272558 24163 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.272660 24295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21809c44bb940ffb007041977ac5696"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.272709 24232 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.272749 24295 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.272794 24295 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.272862 24232 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.273005 26438 raft_consensus.cc:2781] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.273142 26438 raft_consensus.cc:686] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.273259 26438 consensus_queue.cc:237] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.17, Last appended by leader: 17, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.273739 26438 raft_consensus.cc:2781] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.273808 26438 raft_consensus.cc:686] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.273926 26438 consensus_queue.cc:237] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.274152 24295 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.274394 26438 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.274483 26438 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.274585 26438 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.9, Last appended by leader: 9, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.274410 24295 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.277281 24297 raft_consensus.cc:1226] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: []
I20240501 07:59:38.278290 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.278396 24163 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.279044 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.279181 24297 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.279237 24297 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.279438 24163 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.279817 24429 raft_consensus.cc:1261] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240501 07:59:38.280946 25421 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:38.281431 24297 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.281756 24297 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.282286 24295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.282377 24295 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.282429 24295 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.283136 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.283243 24429 raft_consensus.cc:3032] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:38.283280 24429 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.283324 24295 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.283390 24429 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.283516 24429 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.283607 24295 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.284322 24232 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.284440 24429 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.285002 24164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.285115 24164 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 3 FOLLOWER]: Advancing to term 4
W20240501 07:59:38.285123 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.285578 24233 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.285665 25421 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:38.285750 25421 raft_consensus.cc:686] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.285862 25421 consensus_queue.cc:237] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.286082 24164 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.286361 24163 raft_consensus.cc:1261] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240501 07:59:38.286398 25421 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 07:59:38.286463 25421 raft_consensus.cc:686] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.286561 25421 consensus_queue.cc:237] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.286952 26370 consensus_queue.cc:1046] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: INVALID_TERM, Last received: 5.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 07:59:38.287128 26370 consensus_queue.cc:1225] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 6.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20240501 07:59:38.287186 26370 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.287217 26370 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.287322 26370 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 5.18, Last appended by leader: 18, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.287456 26370 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.287715 24297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.287808 24297 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.287853 24297 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.288780 24297 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.289070 24297 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.289284 24299 raft_consensus.cc:1261] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Refusing update from remote peer 1df5867ab7af4bd0a8229838d80e9d7f: Log matching property violated. Preceding OpId in replica: term: 4 index: 25. Preceding OpId from leader: term: 5 index: 27. (index mismatch)
I20240501 07:59:38.289600 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.289700 24428 raft_consensus.cc:3032] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 07:59:38.289731 24428 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.289958 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.290067 24167 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.290517 23894 catalog_manager.cc:5484] T e63f2991bfdf4e4a9aec4534605cf82e P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.290920 24428 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.291097 24428 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 3 FOLLOWER]: Advancing to term 4
I20240501 07:59:38.291111 24167 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.292248 24428 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 4.
I20240501 07:59:38.292755 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.292872 24035 raft_consensus.cc:3032] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.292913 24035 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
W20240501 07:59:38.292973 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.293035 24035 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.14, Last appended by leader: 14, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.293177 24035 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.293537 24232 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.293969 26210 consensus_queue.cc:1035] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:59:38.294113 24035 raft_consensus.cc:2396] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 5 because replica has last-logged OpId of term: 4 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13.
I20240501 07:59:38.294219 26210 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.294293 26210 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.294404 26210 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.294926 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.295075 24035 raft_consensus.cc:2396] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 17.
I20240501 07:59:38.295397 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.295492 24035 raft_consensus.cc:3032] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.295528 24035 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.295640 24035 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.9, Last appended by leader: 9, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.295791 24035 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.296643 24035 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.296866 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.296980 24430 raft_consensus.cc:3032] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.297029 24430 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.297096 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.297173 24034 raft_consensus.cc:3032] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.297205 24034 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.297150 24430 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.297312 24430 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.297317 24034 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.297466 24034 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.298207 24430 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.298305 24034 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.298628 24233 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.299027 26438 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.299109 26438 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.299243 26438 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.300529 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.300631 24167 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.301752 26213 consensus_queue.cc:1035] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20240501 07:59:38.302805 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.303088 26210 consensus_queue.cc:1035] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:59:38.303184 23894 catalog_manager.cc:5484] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.303396 23894 catalog_manager.cc:5484] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 3 to 4, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.303570 23894 catalog_manager.cc:5484] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.303745 23894 catalog_manager.cc:5484] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 3 to 4, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.303905 23894 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.301553 24033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ff4003c1654707a17d65993fb938af" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.304344 24033 raft_consensus.cc:3032] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.304394 24033 raft_consensus.cc:729] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.304543 24033 consensus_queue.cc:260] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.9, Last appended by leader: 9, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.305851 24033 raft_consensus.cc:3037] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.306797 24033 raft_consensus.cc:2396] T 15ff4003c1654707a17d65993fb938af P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240501 07:59:38.307193 24167 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
W20240501 07:59:38.308456 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.308660 24430 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20240501 07:59:38.309057 26438 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 07:59:38.311790 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.315312 24163 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20240501 07:59:38.315956 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.316506 24296 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.318185 24296 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.316697 26438 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 07:59:38.319842 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.320937 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.322722 24232 leader_election.cc:302] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d371a71c02a4432aca2fed33b2545d5; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888
I20240501 07:59:38.323204 26210 raft_consensus.cc:2726] T 15ff4003c1654707a17d65993fb938af P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 07:59:38.324365 24165 raft_consensus.cc:1226] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: [5.17-5.17]
I20240501 07:59:38.327152 23893 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.327389 23893 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:38.331905 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.333334 24296 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.333851 24296 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.334229 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.334339 24167 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.335789 24167 raft_consensus.cc:2454] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.336057 24232 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.337860 26443 raft_consensus.cc:2781] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.339851 24296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
W20240501 07:59:38.340387 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57228: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.340082 24296 raft_consensus.cc:482] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.341207 24296 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.342442 26443 raft_consensus.cc:686] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.342404 24296 raft_consensus.cc:504] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.343569 26443 consensus_queue.cc:237] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.343942 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.344058 24428 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.345052 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.345144 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.345263 24036 raft_consensus.cc:3032] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.345317 24036 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.345489 24036 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.345657 24036 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.347148 24428 raft_consensus.cc:2454] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 6.
I20240501 07:59:38.347836 24036 raft_consensus.cc:2396] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240501 07:59:38.349174 24233 leader_election.cc:302] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.349365 26444 raft_consensus.cc:2781] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.349517 26444 raft_consensus.cc:686] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.349757 26444 consensus_queue.cc:237] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.349951 23893 catalog_manager.cc:5484] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.353969 24296 leader_election.cc:288] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 6 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.354504 24299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
from {username='slave'} at 127.0.0.1:57332
I20240501 07:59:38.354609 24299 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.354662 24299 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.354947 24429 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20240501 07:59:38.355494 26210 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:38.355844 24299 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.356290 24299 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
W20240501 07:59:38.356766 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.363054 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.363245 24163 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.364375 24163 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.365185 24163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.365352 24163 raft_consensus.cc:482] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.365535 24163 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.366888 23890 catalog_manager.cc:5484] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 6 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.367870 24232 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.368053 24296 raft_consensus.cc:1226] T 130aef27115540ed8e0bc20158c1aeea P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: [5.35-5.35]
I20240501 07:59:38.368286 26443 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.368378 26443 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Leader. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.368494 26443 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.368891 24426 raft_consensus.cc:1226] T 130aef27115540ed8e0bc20158c1aeea P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Rejecting Update request from peer 1df5867ab7af4bd0a8229838d80e9d7f for earlier term 5. Current term is 6. Ops: [5.35-5.35]
I20240501 07:59:38.370146 26213 consensus_queue.cc:1046] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: INVALID_TERM, Last received: 5.34, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20240501 07:59:38.370646 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.373036 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.373144 24034 raft_consensus.cc:3032] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.373185 24034 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.373307 24034 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.373459 24034 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.374421 24034 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d371a71c02a4432aca2fed33b2545d5 in term 5.
I20240501 07:59:38.374879 24033 raft_consensus.cc:1261] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
W20240501 07:59:38.376581 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.376991 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130aef27115540ed8e0bc20158c1aeea" candidate_uuid: "9d371a71c02a4432aca2fed33b2545d5" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.377702 26213 consensus_queue.cc:1225] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer 9d371a71c02a4432aca2fed33b2545d5 log is divergent from this leader: its last log entry 6.35 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 34
I20240501 07:59:38.377771 26370 consensus_queue.cc:1046] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: INVALID_TERM, Last received: 5.34, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 07:59:38.378095 24036 raft_consensus.cc:3032] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.378129 24036 raft_consensus.cc:729] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.378103 23890 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 5, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131). New cstate: current_term: 5 leader_uuid: "9d371a71c02a4432aca2fed33b2545d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.378235 24036 consensus_queue.cc:260] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.35, Last appended by leader: 35, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.378357 24036 raft_consensus.cc:3037] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.379369 24036 raft_consensus.cc:2396] T 130aef27115540ed8e0bc20158c1aeea P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d371a71c02a4432aca2fed33b2545d5 for term 6 because replica has last-logged OpId of term: 5 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 34.
I20240501 07:59:38.382771 26443 consensus_queue.cc:1035] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 07:59:38.383176 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.383608 24163 raft_consensus.cc:504] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.383795 24166 raft_consensus.cc:1226] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Rejecting Update request from peer 9d371a71c02a4432aca2fed33b2545d5 for earlier term 6. Current term is 7. Ops: [6.14-6.14]
I20240501 07:59:38.384685 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.385087 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6523f8bf6c574f36bd31be66abf6fd23" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.385185 24296 raft_consensus.cc:3032] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.385223 24296 raft_consensus.cc:729] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.385267 24163 leader_election.cc:288] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.385350 24296 consensus_queue.cc:260] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 6.14, Last appended by leader: 14, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.385457 24296 raft_consensus.cc:3037] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.385913 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.386008 24165 raft_consensus.cc:482] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.386051 24165 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.386399 24296 raft_consensus.cc:2396] T 6523f8bf6c574f36bd31be66abf6fd23 P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 7 because replica has last-logged OpId of term: 6 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 13.
I20240501 07:59:38.386760 24104 leader_election.cc:302] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.386932 24165 raft_consensus.cc:504] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.387027 26309 raft_consensus.cc:2726] T 6523f8bf6c574f36bd31be66abf6fd23 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 07:59:38.387447 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.387538 24296 raft_consensus.cc:3032] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.387596 24296 raft_consensus.cc:729] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.387703 24296 consensus_queue.cc:260] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 5.20, Last appended by leader: 20, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.387796 24296 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.387849 24165 leader_election.cc:288] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.388593 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af824adcc5ae43e79679d0257a4e578f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.388679 24296 raft_consensus.cc:2396] T af824adcc5ae43e79679d0257a4e578f P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 19.
I20240501 07:59:38.388685 24036 raft_consensus.cc:3037] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.389389 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.389672 24036 raft_consensus.cc:2454] T af824adcc5ae43e79679d0257a4e578f P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.389922 24101 leader_election.cc:302] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.390125 26309 raft_consensus.cc:2781] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.390208 26309 raft_consensus.cc:686] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.390316 26309 consensus_queue.cc:237] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.395721 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.396399 23890 catalog_manager.cc:5484] T af824adcc5ae43e79679d0257a4e578f P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.396821 24165 raft_consensus.cc:482] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.396903 24165 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.398038 24165 raft_consensus.cc:504] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.398921 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.399042 24296 raft_consensus.cc:3032] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.399085 24296 raft_consensus.cc:729] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.399202 24296 consensus_queue.cc:260] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.399323 24296 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.399482 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82547c4a0a8475b9e9a3aaf5a486438" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.399571 24036 raft_consensus.cc:3037] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.400452 24036 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.400657 24296 raft_consensus.cc:2454] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.400938 24101 leader_election.cc:302] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.401089 26309 raft_consensus.cc:2781] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.401158 26309 raft_consensus.cc:686] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.401259 26309 consensus_queue.cc:237] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
W20240501 07:59:38.401499 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.402168 24165 leader_election.cc:288] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.402712 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.402982 24165 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.403069 24165 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.403954 24165 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.404472 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.404567 24429 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:38.405177 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.405458 24429 raft_consensus.cc:2396] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 5 because replica has last-logged OpId of term: 4 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 20.
W20240501 07:59:38.405855 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.406657 24163 raft_consensus.cc:1226] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Rejecting Update request from peer 9d371a71c02a4432aca2fed33b2545d5 for earlier term 4. Current term is 5. Ops: [4.21-4.21]
I20240501 07:59:38.408393 26210 consensus_queue.cc:1046] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: INVALID_TERM, Last received: 4.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 07:59:38.408586 26210 raft_consensus.cc:3032] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.408635 26210 raft_consensus.cc:729] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.408759 26210 consensus_queue.cc:260] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 4.21, Last appended by leader: 21, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.408921 26210 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.409106 23893 catalog_manager.cc:5484] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:38.409297 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.410897 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
W20240501 07:59:38.411151 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57164: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.411315 24296 raft_consensus.cc:2396] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 5 because replica has last-logged OpId of term: 4 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 20.
I20240501 07:59:38.411473 24165 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.411629 24104 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.412007 25335 raft_consensus.cc:2726] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 07:59:38.412268 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.412432 24165 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.412546 24165 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.413375 24165 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.413712 24165 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.414414 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.414515 24165 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.414574 24165 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.415289 24165 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.416052 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.416101 24034 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
I20240501 07:59:38.416172 24427 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.416292 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.416391 24299 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.416427 24299 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.416543 24299 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.19, Last appended by leader: 19, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.416550 26309 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 07:59:38.416651 24299 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.417209 24427 raft_consensus.cc:2454] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.417471 24102 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.417646 24299 raft_consensus.cc:2396] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 18.
I20240501 07:59:38.417649 26309 raft_consensus.cc:2781] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.417716 26309 raft_consensus.cc:686] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.417836 26309 consensus_queue.cc:237] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.423254 24296 raft_consensus.cc:1261] T c82547c4a0a8475b9e9a3aaf5a486438 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
I20240501 07:59:38.423574 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.423702 24036 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.425199 24165 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.427592 24036 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.428241 24101 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.428573 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.428689 24429 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.428582 26309 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.428870 26309 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.429188 26309 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.429402 23890 catalog_manager.cc:5484] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.430470 24429 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
W20240501 07:59:38.433120 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.433873 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49982: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.436575 26309 consensus_queue.cc:1035] T c82547c4a0a8475b9e9a3aaf5a486438 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240501 07:59:38.436985 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.437899 23890 catalog_manager.cc:5484] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 6, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
W20240501 07:59:38.439731 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.442993 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.448596 24163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.449668 24163 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.449802 24163 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:38.450364 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.450940 24163 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.451659 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.451786 24296 raft_consensus.cc:3032] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.451829 24296 raft_consensus.cc:729] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.451982 24296 consensus_queue.cc:260] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 4.20, Last appended by leader: 20, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.452160 24296 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.452155 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.452421 24429 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.453207 24296 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.453527 24104 leader_election.cc:302] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.453776 26309 raft_consensus.cc:2781] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.453855 26309 raft_consensus.cc:686] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.453992 26309 consensus_queue.cc:237] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 4.20, Last appended by leader: 20, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
W20240501 07:59:38.454218 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.455571 24429 raft_consensus.cc:2454] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.455957 24163 leader_election.cc:288] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.456627 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.456785 24165 raft_consensus.cc:482] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.456873 24165 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.456991 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.457731 23890 catalog_manager.cc:5484] T 8fb726fc00544504a34a37bf3226f889 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.457854 24165 raft_consensus.cc:504] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.458554 24165 leader_election.cc:288] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.458891 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.459054 24036 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
W20240501 07:59:38.459336 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50024: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.459554 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bd1d3faec840c5baf0454c33f8130f" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.459872 24429 raft_consensus.cc:3037] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.460022 24166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21809c44bb940ffb007041977ac5696"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.460124 24036 raft_consensus.cc:2454] T 48bd1d3faec840c5baf0454c33f8130f P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.460242 24166 raft_consensus.cc:482] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.460366 24166 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.460896 24101 leader_election.cc:302] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.461135 25335 raft_consensus.cc:2781] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.461210 25335 raft_consensus.cc:686] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.461328 25335 consensus_queue.cc:237] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.461827 24429 raft_consensus.cc:2396] T 48bd1d3faec840c5baf0454c33f8130f P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 4 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
W20240501 07:59:38.462159 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.463196 24166 raft_consensus.cc:504] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
W20240501 07:59:38.463394 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.463522 24166 leader_election.cc:288] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.463963 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.464075 24296 raft_consensus.cc:3032] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.464118 24296 raft_consensus.cc:729] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.464221 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21809c44bb940ffb007041977ac5696" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.464310 24429 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.464260 24296 consensus_queue.cc:260] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.9, Last appended by leader: 9, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.464411 24296 raft_consensus.cc:3037] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.464491 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.464602 24165 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.464692 24165 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.465236 24429 raft_consensus.cc:2454] T b21809c44bb940ffb007041977ac5696 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.465351 24296 raft_consensus.cc:2396] T b21809c44bb940ffb007041977ac5696 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 5 because replica has last-logged OpId of term: 4 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8.
I20240501 07:59:38.465495 24102 leader_election.cc:302] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.465662 26309 raft_consensus.cc:2781] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.465739 26309 raft_consensus.cc:686] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.465862 26309 consensus_queue.cc:237] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.465875 24165 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.466223 24165 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.466387 23890 catalog_manager.cc:5484] T 48bd1d3faec840c5baf0454c33f8130f P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 6, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.466881 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.466984 24427 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.467046 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.466997 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.467141 24165 raft_consensus.cc:482] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.467188 24165 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.467185 24296 raft_consensus.cc:3032] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 07:59:38.467221 24296 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.467335 24296 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.19, Last appended by leader: 19, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.467463 24296 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.467917 24427 raft_consensus.cc:2454] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.468152 24165 raft_consensus.cc:504] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.468372 24296 raft_consensus.cc:2396] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 5 because replica has last-logged OpId of term: 4 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 18.
I20240501 07:59:38.468935 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.468982 24165 leader_election.cc:288] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.469069 24036 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.468979 24102 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.469349 25335 raft_consensus.cc:2781] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.469416 25335 raft_consensus.cc:686] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.469521 25335 consensus_queue.cc:237] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.469944 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.470046 24165 raft_consensus.cc:482] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.470100 24165 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.470103 24036 raft_consensus.cc:2454] T 3f4f9ceedafe4c90ac182f1ba61d583a P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 7.
I20240501 07:59:38.470736 24101 leader_election.cc:302] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.470798 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f9ceedafe4c90ac182f1ba61d583a" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.470886 26309 raft_consensus.cc:2781] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 07:59:38.470906 24296 raft_consensus.cc:3032] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.470944 24296 raft_consensus.cc:729] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.470948 26309 raft_consensus.cc:686] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [term 7 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.471079 24296 consensus_queue.cc:260] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.471076 26309 consensus_queue.cc:237] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.471210 24296 raft_consensus.cc:3037] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.471359 24165 raft_consensus.cc:504] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.471450 23890 catalog_manager.cc:5484] T b21809c44bb940ffb007041977ac5696 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.471676 23890 catalog_manager.cc:5484] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 5, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.472101 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
W20240501 07:59:38.472182 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.472332 24165 leader_election.cc:288] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.472205 24299 raft_consensus.cc:3032] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.472498 24299 raft_consensus.cc:729] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.472656 24296 raft_consensus.cc:2396] T 3f4f9ceedafe4c90ac182f1ba61d583a P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 07:59:38.472661 24299 consensus_queue.cc:260] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 5.14, Last appended by leader: 14, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.473065 24299 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.473232 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133202f01ac942f9af1fe2d0229f38b2" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.473349 24036 raft_consensus.cc:3037] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.474143 24299 raft_consensus.cc:2396] T 133202f01ac942f9af1fe2d0229f38b2 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13.
I20240501 07:59:38.474366 24036 raft_consensus.cc:2396] T 133202f01ac942f9af1fe2d0229f38b2 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 4 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13.
I20240501 07:59:38.474735 24101 leader_election.cc:302] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.474889 25335 raft_consensus.cc:2726] T 133202f01ac942f9af1fe2d0229f38b2 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240501 07:59:38.475217 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.475244 23890 catalog_manager.cc:5484] T 3f4f9ceedafe4c90ac182f1ba61d583a P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 6 to 7, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 7 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } }
I20240501 07:59:38.475677 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.475806 24165 raft_consensus.cc:482] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.475893 24165 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.476905 24165 raft_consensus.cc:504] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.477486 24165 leader_election.cc:288] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.477670 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.477784 24036 raft_consensus.cc:3032] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.477850 24036 raft_consensus.cc:729] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.478013 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5d8581db864dd89b7e292cb93670c1" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.478016 24036 consensus_queue.cc:260] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 5.21, Last appended by leader: 21, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.478130 24429 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.478209 24036 raft_consensus.cc:3037] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.478885 24429 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.478899 24036 raft_consensus.cc:2454] T 9e5d8581db864dd89b7e292cb93670c1 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.479175 24102 leader_election.cc:302] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.479333 25335 raft_consensus.cc:2781] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.479449 25335 raft_consensus.cc:686] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.479593 25335 consensus_queue.cc:237] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 5.21, Last appended by leader: 21, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.480432 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.480580 24165 raft_consensus.cc:482] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.480671 24165 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.481503 23890 catalog_manager.cc:5484] T 9e5d8581db864dd89b7e292cb93670c1 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.482528 24165 raft_consensus.cc:504] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.483371 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.483479 24036 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.483474 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80205bf3beff4e3bba804332a6aa8c73" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.483579 24299 raft_consensus.cc:3032] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.483629 24299 raft_consensus.cc:729] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.483777 24299 consensus_queue.cc:260] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.483922 24299 raft_consensus.cc:3037] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.484402 24036 raft_consensus.cc:2396] T 80205bf3beff4e3bba804332a6aa8c73 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 07:59:38.485000 24299 raft_consensus.cc:2396] T 80205bf3beff4e3bba804332a6aa8c73 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20240501 07:59:38.485766 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.485749 24104 leader_election.cc:302] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.485951 26309 raft_consensus.cc:2726] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 07:59:38.486048 24165 leader_election.cc:288] T 80205bf3beff4e3bba804332a6aa8c73 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.487380 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.487504 24165 raft_consensus.cc:482] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.487567 24165 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 4 FOLLOWER]: Advancing to term 5
W20240501 07:59:38.488391 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.488621 24165 raft_consensus.cc:504] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.489077 24165 leader_election.cc:288] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.489384 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.489403 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14500c7bb754282a0717ccd4257f6eb" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 5 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.489514 24036 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.489513 24429 raft_consensus.cc:3037] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 4 FOLLOWER]: Advancing to term 5
I20240501 07:59:38.490372 24036 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.490624 24429 raft_consensus.cc:2454] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 5.
I20240501 07:59:38.490713 24101 leader_election.cc:302] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.490913 26309 raft_consensus.cc:2781] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 07:59:38.491025 26309 raft_consensus.cc:686] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.491173 26309 consensus_queue.cc:237] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.15, Last appended by leader: 15, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.492352 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.492561 24165 raft_consensus.cc:482] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
W20240501 07:59:38.492573 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.492609 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57246: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.492725 24165 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.493172 23893 catalog_manager.cc:5484] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 3 to 5, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 5 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
W20240501 07:59:38.495601 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.493973 24165 raft_consensus.cc:504] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.496824 24165 leader_election.cc:288] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.497066 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.497151 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6313ef646004e3eba64205bd4f535ce" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.497187 24036 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.497249 24429 raft_consensus.cc:3037] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.498271 24429 raft_consensus.cc:2396] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 4 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240501 07:59:38.498252 24036 raft_consensus.cc:2454] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.498531 24101 leader_election.cc:302] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.498765 25335 raft_consensus.cc:2781] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.498874 25335 raft_consensus.cc:686] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.499082 25335 consensus_queue.cc:237] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.501485 23893 catalog_manager.cc:5484] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 4 to 6, leader changed from 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
W20240501 07:59:38.506436 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.498548 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.507159 24165 raft_consensus.cc:482] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.507227 24165 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.508239 24165 raft_consensus.cc:504] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.508968 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.509114 24299 raft_consensus.cc:3032] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.509184 24299 raft_consensus.cc:729] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.509328 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af81850415b45bd9f8c1d52044bd372" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.509346 24299 consensus_queue.cc:260] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.18, Last appended by leader: 18, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.509438 24036 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.509500 24299 raft_consensus.cc:3037] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.510462 24036 raft_consensus.cc:2396] T 6af81850415b45bd9f8c1d52044bd372 P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 7 because replica has last-logged OpId of term: 5 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 16.
I20240501 07:59:38.510561 24299 raft_consensus.cc:2396] T 6af81850415b45bd9f8c1d52044bd372 P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 7 because replica has last-logged OpId of term: 6 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 16.
I20240501 07:59:38.510692 24165 leader_election.cc:288] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.510987 24101 leader_election.cc:302] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474; no voters: 1df5867ab7af4bd0a8229838d80e9d7f, 9d371a71c02a4432aca2fed33b2545d5
W20240501 07:59:38.511483 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53218: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.512051 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.512143 24165 raft_consensus.cc:482] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.512192 24165 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.512910 24165 raft_consensus.cc:504] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.513180 24165 leader_election.cc:288] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.513693 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.513794 24036 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.513950 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.514035 24165 raft_consensus.cc:482] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.514071 24165 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.514626 24036 raft_consensus.cc:2454] T 92db375d17784d2197a1f0f427e2d430 P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 7.
I20240501 07:59:38.514662 24165 raft_consensus.cc:504] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.514932 24101 leader_election.cc:302] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.515048 25335 raft_consensus.cc:2726] T 6af81850415b45bd9f8c1d52044bd372 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 07:59:38.515129 26309 raft_consensus.cc:2781] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 07:59:38.515209 26309 raft_consensus.cc:686] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.515267 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.515365 24296 raft_consensus.cc:3032] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.515326 26309 consensus_queue.cc:237] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.9, Last appended by leader: 9, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.515401 24296 raft_consensus.cc:729] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.515518 24296 consensus_queue.cc:260] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.515645 24296 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.516563 24296 raft_consensus.cc:2396] T e786d8c542154add8bf34774a6ef095d P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240501 07:59:38.516932 24165 leader_election.cc:288] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
W20240501 07:59:38.516974 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.517489 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e786d8c542154add8bf34774a6ef095d" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.517608 24429 raft_consensus.cc:3037] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.518273 23889 catalog_manager.cc:5484] T 92db375d17784d2197a1f0f427e2d430 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 6 to 7, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 7 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.518558 24429 raft_consensus.cc:2454] T e786d8c542154add8bf34774a6ef095d P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.518846 24102 leader_election.cc:302] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.519091 26309 raft_consensus.cc:2781] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.519165 26309 raft_consensus.cc:686] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.519268 26309 consensus_queue.cc:237] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.519443 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
W20240501 07:59:38.519497 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.519533 24165 raft_consensus.cc:482] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.519580 24165 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.520431 24165 raft_consensus.cc:504] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.520829 24165 leader_election.cc:288] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.521386 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.521490 24429 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.522424 24429 raft_consensus.cc:2454] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.522838 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f318e2ba0c4a26ad4cd8630bb0748c" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.522958 24296 raft_consensus.cc:3032] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.523046 24296 raft_consensus.cc:729] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.523103 24102 leader_election.cc:302] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.523175 24296 consensus_queue.cc:260] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.523298 25335 raft_consensus.cc:2781] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.523313 24296 raft_consensus.cc:3037] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.523373 25335 raft_consensus.cc:686] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.523483 25335 consensus_queue.cc:237] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.524312 24296 raft_consensus.cc:2396] T e1f318e2ba0c4a26ad4cd8630bb0748c P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240501 07:59:38.524663 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db375d17784d2197a1f0f427e2d430" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 7 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.524787 24299 raft_consensus.cc:3032] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.524847 24299 raft_consensus.cc:729] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.524883 23895 catalog_manager.cc:5484] T e786d8c542154add8bf34774a6ef095d P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.525007 24299 consensus_queue.cc:260] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.525169 24299 raft_consensus.cc:3037] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.526189 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.526289 24299 raft_consensus.cc:2396] T 92db375d17784d2197a1f0f427e2d430 P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3748d2c0aeb740f98f8c2caabad21474 for term 7 because replica has last-logged OpId of term: 6 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 9.
I20240501 07:59:38.526284 24165 raft_consensus.cc:482] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.526335 24165 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.527915 24429 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
I20240501 07:59:38.527895 24165 raft_consensus.cc:504] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.528064 24036 raft_consensus.cc:1261] T b14500c7bb754282a0717ccd4257f6eb P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
I20240501 07:59:38.528270 26309 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 07:59:38.528535 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.528652 24036 raft_consensus.cc:3032] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.528712 24036 raft_consensus.cc:729] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.528842 24036 consensus_queue.cc:260] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 5.22, Last appended by leader: 22, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.529008 24036 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.529947 24036 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.531183 24101 leader_election.cc:302] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.531232 25335 consensus_queue.cc:1035] T b14500c7bb754282a0717ccd4257f6eb P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:59:38.531617 25335 raft_consensus.cc:2781] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.531708 25335 raft_consensus.cc:686] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.531842 25335 consensus_queue.cc:237] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 5.22, Last appended by leader: 22, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.531839 23895 catalog_manager.cc:5484] T e1f318e2ba0c4a26ad4cd8630bb0748c P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.532675 26177 mvcc.cc:205] Tried to move back new op lower bound from 7022798350444306432 to 7022798350301335552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798350444306432 or (T in {7022798350444306432})}]
W20240501 07:59:38.533643 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53298: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.534407 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.535512 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57316: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
W20240501 07:59:38.537811 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.538105 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9401efd19547c2ac46a58c72c25834" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.538254 24429 raft_consensus.cc:3037] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.539384 24429 raft_consensus.cc:2454] T 9e9401efd19547c2ac46a58c72c25834 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.539750 24165 leader_election.cc:288] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159)
I20240501 07:59:38.540947 24165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0272069f81436f81725c3092e0e382"
dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
from {username='slave'} at 127.0.0.1:50000
I20240501 07:59:38.541146 24165 raft_consensus.cc:482] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.541258 24165 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.542294 24165 raft_consensus.cc:504] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.542443 23893 catalog_manager.cc:5484] T 9e9401efd19547c2ac46a58c72c25834 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } }
I20240501 07:59:38.544287 24429 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 4 index: 18. Preceding OpId from leader: term: 6 index: 20. (index mismatch)
I20240501 07:59:38.544776 25335 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240501 07:59:38.545245 24036 raft_consensus.cc:1261] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 20. (index mismatch)
I20240501 07:59:38.546418 24036 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20240501 07:59:38.547070 24427 raft_consensus.cc:1261] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Refusing update from remote peer 3748d2c0aeb740f98f8c2caabad21474: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20240501 07:59:38.547572 25335 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:38.547685 25335 consensus_queue.cc:1225] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Peer 3b8f81da8ae74d3687ef72a2c9ce0888 log is divergent from this leader: its last log entry 4.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240501 07:59:38.547832 26309 consensus_queue.cc:1035] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 07:59:38.549145 25335 consensus_queue.cc:1035] T c6313ef646004e3eba64205bd4f535ce P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:38.551335 24429 pending_rounds.cc:76] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Aborting all ops after (but not including) 9
I20240501 07:59:38.551388 24429 pending_rounds.cc:97] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888: Aborting uncommitted NO_OP operation due to leader change: 4.10
I20240501 07:59:38.551417 24429 raft_consensus.cc:2866] T c6313ef646004e3eba64205bd4f535ce P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 07:59:38.551913 24127 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50082: Illegal state: replica 3748d2c0aeb740f98f8c2caabad21474 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.552891 24165 leader_election.cc:288] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Requested vote from peers 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132:43159), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.554428 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
I20240501 07:59:38.554600 24429 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.555598 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0272069f81436f81725c3092e0e382" candidate_uuid: "3748d2c0aeb740f98f8c2caabad21474" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.555716 24036 raft_consensus.cc:3032] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.555763 24036 raft_consensus.cc:729] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1df5867ab7af4bd0a8229838d80e9d7f, State: Running, Role: LEADER
I20240501 07:59:38.555907 24036 consensus_queue.cc:260] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.555987 24429 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.556098 24036 raft_consensus.cc:3037] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.557140 24036 raft_consensus.cc:2454] T 4f0272069f81436f81725c3092e0e382 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3748d2c0aeb740f98f8c2caabad21474 in term 6.
I20240501 07:59:38.557670 24101 leader_election.cc:302] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3748d2c0aeb740f98f8c2caabad21474; no voters:
I20240501 07:59:38.557847 26309 raft_consensus.cc:2781] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.557941 26309 raft_consensus.cc:686] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Leader. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.558048 26309 consensus_queue.cc:237] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.561995 24426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.562099 24426 raft_consensus.cc:482] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.562152 24426 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Advancing to term 7
W20240501 07:59:38.563118 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53198: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.563405 24426 raft_consensus.cc:504] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.563750 24426 leader_election.cc:288] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.564569 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.564698 24299 raft_consensus.cc:3032] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.564745 24299 raft_consensus.cc:729] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.564913 24299 consensus_queue.cc:260] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 6.11, Last appended by leader: 11, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.565078 24299 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.566066 24299 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 7.
I20240501 07:59:38.566581 24366 leader_election.cc:302] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888, 9d371a71c02a4432aca2fed33b2545d5; no voters:
I20240501 07:59:38.566735 26369 raft_consensus.cc:2781] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 07:59:38.566812 26369 raft_consensus.cc:686] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.566941 26369 consensus_queue.cc:237] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 6.11, Last appended by leader: 11, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.568706 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebcc0e2a36945c4ad094dc7a8df62bf" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.568871 24034 raft_consensus.cc:3037] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.569044 24426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.569168 24426 raft_consensus.cc:482] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.569219 24426 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.570070 24034 raft_consensus.cc:2454] T 2ebcc0e2a36945c4ad094dc7a8df62bf P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 7.
I20240501 07:59:38.570235 24426 raft_consensus.cc:504] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.570574 24426 leader_election.cc:288] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
W20240501 07:59:38.571405 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57192: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
W20240501 07:59:38.571882 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57312: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.572145 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.572250 24034 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.572315 24427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.572404 24427 raft_consensus.cc:482] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.572453 24427 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.573181 24034 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 7.
I20240501 07:59:38.573365 24427 raft_consensus.cc:504] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.573697 24427 leader_election.cc:288] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Requested vote from peers 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707), 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575)
I20240501 07:59:38.574446 24427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1056fad606894561bd417d1586cf8032"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.574537 24427 raft_consensus.cc:482] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.574605 24427 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.575047 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.575152 24296 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.575798 24427 raft_consensus.cc:504] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.576180 24296 raft_consensus.cc:2396] T 4da90005c90d4944996a8a2d2bf59c18 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 6 because replica has last-logged OpId of term: 4 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 18.
I20240501 07:59:38.576447 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1ffb7f2be64f05ba2fe6277b60f245" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.576545 24299 raft_consensus.cc:3032] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.576583 24299 raft_consensus.cc:729] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.576576 24427 leader_election.cc:288] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.576719 24299 consensus_queue.cc:260] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.576857 24363 leader_election.cc:302] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.576900 24299 raft_consensus.cc:3037] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.577293 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.577397 24163 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.577447 24034 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
I20240501 07:59:38.577775 26210 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 07:59:38.577894 26210 consensus_queue.cc:1225] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Peer 1df5867ab7af4bd0a8229838d80e9d7f log is divergent from this leader: its last log entry 5.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20240501 07:59:38.577929 24427 raft_consensus.cc:1261] T 090ee3c1fc884691810492f5eccbef67 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Refusing update from remote peer 9d371a71c02a4432aca2fed33b2545d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
I20240501 07:59:38.578161 26221 raft_consensus.cc:2781] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 07:59:38.578238 26221 raft_consensus.cc:686] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.578369 26221 consensus_queue.cc:237] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.578414 24163 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 6.
I20240501 07:59:38.578570 24034 pending_rounds.cc:76] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting all ops after (but not including) 15
I20240501 07:59:38.578608 24034 pending_rounds.cc:97] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f: Aborting uncommitted NO_OP operation due to leader change: 5.16
I20240501 07:59:38.578634 24034 raft_consensus.cc:2866] T 090ee3c1fc884691810492f5eccbef67 P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 07:59:38.578874 24363 leader_election.cc:302] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3748d2c0aeb740f98f8c2caabad21474, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.577769 24299 raft_consensus.cc:2454] T 7a1ffb7f2be64f05ba2fe6277b60f245 P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 7.
I20240501 07:59:38.579144 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.579233 24430 raft_consensus.cc:482] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.579284 24430 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.579299 26210 consensus_queue.cc:1035] T 090ee3c1fc884691810492f5eccbef67 P 9d371a71c02a4432aca2fed33b2545d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 07:59:38.579668 26369 raft_consensus.cc:2781] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.579737 26369 raft_consensus.cc:686] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.579838 26369 consensus_queue.cc:237] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 4.21, Last appended by leader: 21, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } }
I20240501 07:59:38.580181 24430 raft_consensus.cc:504] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.580466 24430 leader_election.cc:288] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675)
I20240501 07:59:38.581184 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.581282 24036 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.582267 24036 raft_consensus.cc:2454] T 3119928e27b14eb0a68ae0aa0b70059c P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 6.
I20240501 07:59:38.583602 24296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1056fad606894561bd417d1586cf8032" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.583711 24296 raft_consensus.cc:3037] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.584657 24296 raft_consensus.cc:2454] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 6.
I20240501 07:59:38.584936 24363 leader_election.cc:302] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
I20240501 07:59:38.584986 24427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.585078 24427 raft_consensus.cc:482] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.585120 24427 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.585907 24427 raft_consensus.cc:504] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.586123 26369 raft_consensus.cc:2781] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 07:59:38.586186 24427 leader_election.cc:288] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Requested vote from peers 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129:37675), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.586232 26369 raft_consensus.cc:686] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.586357 26369 consensus_queue.cc:237] T 3119928e27b14eb0a68ae0aa0b70059c P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.14, Last appended by leader: 14, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.586623 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90005c90d4944996a8a2d2bf59c18" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.586726 24165 raft_consensus.cc:3032] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.586766 24165 raft_consensus.cc:729] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.586879 24165 consensus_queue.cc:260] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.19, Last appended by leader: 19, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } }
I20240501 07:59:38.586969 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf9d5a354ac4ef8a8df43802063eb4e" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: true dest_uuid: "1df5867ab7af4bd0a8229838d80e9d7f"
I20240501 07:59:38.587035 24165 raft_consensus.cc:3037] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.587081 24036 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.587924 24036 raft_consensus.cc:2454] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b8f81da8ae74d3687ef72a2c9ce0888 in term 7.
I20240501 07:59:38.588035 24165 raft_consensus.cc:2396] T 4da90005c90d4944996a8a2d2bf59c18 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 6 because replica has last-logged OpId of term: 5 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 18.
I20240501 07:59:38.588475 24363 leader_election.cc:302] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.588685 24363 leader_election.cc:302] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df5867ab7af4bd0a8229838d80e9d7f, 3b8f81da8ae74d3687ef72a2c9ce0888; no voters:
W20240501 07:59:38.589080 23996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57198: Illegal state: replica 1df5867ab7af4bd0a8229838d80e9d7f is not leader of this config: current role FOLLOWER
I20240501 07:59:38.589037 23893 catalog_manager.cc:5484] T 4f0272069f81436f81725c3092e0e382 P 3748d2c0aeb740f98f8c2caabad21474 reported cstate change: term changed from 5 to 6, leader changed from 1df5867ab7af4bd0a8229838d80e9d7f (127.0.99.129) to 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130). New cstate: current_term: 6 leader_uuid: "3748d2c0aeb740f98f8c2caabad21474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } }
I20240501 07:59:38.589385 26221 raft_consensus.cc:2726] T 4da90005c90d4944996a8a2d2bf59c18 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 07:59:38.589464 26221 raft_consensus.cc:2781] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 07:59:38.589514 26221 raft_consensus.cc:686] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 LEADER]: Becoming Leader. State: Replica: 3b8f81da8ae74d3687ef72a2c9ce0888, State: Running, Role: LEADER
I20240501 07:59:38.589617 26221 consensus_queue.cc:237] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.592725 24163 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 4 index: 20. Preceding OpId from leader: term: 6 index: 23. (index mismatch)
I20240501 07:59:38.593317 26481 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
W20240501 07:59:38.593506 24390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53234: Illegal state: replica 3b8f81da8ae74d3687ef72a2c9ce0888 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.593814 24427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.593909 24427 raft_consensus.cc:482] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.593956 24427 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.594933 24427 raft_consensus.cc:504] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.595265 24427 leader_election.cc:288] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Requested vote from peers 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131:43575), 3748d2c0aeb740f98f8c2caabad21474 (127.0.99.130:40707)
I20240501 07:59:38.595639 24297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.595739 24297 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.596203 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3119928e27b14eb0a68ae0aa0b70059c" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 6 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "9d371a71c02a4432aca2fed33b2545d5"
I20240501 07:59:38.596441 24299 raft_consensus.cc:3032] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 07:59:38.596498 24299 raft_consensus.cc:729] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9d371a71c02a4432aca2fed33b2545d5, State: Running, Role: LEADER
I20240501 07:59:38.596659 24299 consensus_queue.cc:260] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 5.15, Last appended by leader: 15, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } }
I20240501 07:59:38.596715 24297 raft_consensus.cc:2396] T 02c7ed9d33204e06a2579bde5ca71960 P 9d371a71c02a4432aca2fed33b2545d5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 7 because replica has last-logged OpId of term: 5 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 18.
I20240501 07:59:38.596778 24299 raft_consensus.cc:3037] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.598093 24166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c7ed9d33204e06a2579bde5ca71960" candidate_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" candidate_term: 7 candidate_status { last_received { term: 4 index: 18 } } ignore_live_leader: true dest_uuid: "3748d2c0aeb740f98f8c2caabad21474"
I20240501 07:59:38.598186 24166 raft_consensus.cc:3032] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.598223 24166 raft_consensus.cc:729] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.598345 24166 consensus_queue.cc:260] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.19, Last appended by leader: 19, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.598477 24166 raft_consensus.cc:3037] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.599509 24166 raft_consensus.cc:2396] T 02c7ed9d33204e06a2579bde5ca71960 P 3748d2c0aeb740f98f8c2caabad21474 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 7 because replica has last-logged OpId of term: 6 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 18.
I20240501 07:59:38.600239 24296 raft_consensus.cc:1261] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Refusing update from remote peer 3b8f81da8ae74d3687ef72a2c9ce0888: Log matching property violated. Preceding OpId in replica: term: 4 index: 21. Preceding OpId from leader: term: 6 index: 23. (index mismatch)
I20240501 07:59:38.600636 26369 consensus_queue.cc:1035] T 1056fad606894561bd417d1586cf8032 P 3b8f81da8ae74d3687ef72a2c9ce0888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20240501 07:59:38.600809 24033 raft_consensus.cc:1226] T ebf9d5a354ac4ef8a8df43802063eb4e P 1df5867ab7af4bd0a8229838d80e9d7f [term 7 FOLLOWER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 6. Current term is 7. Ops: [6.21-6.21]
I20240501 07:59:38.601150 26309 consensus_queue.cc:1046] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 }, Status: INVALID_TERM, Last received: 6.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 07:59:38.601356 26309 raft_consensus.cc:3032] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 07:59:38.601401 26309 raft_consensus.cc:729] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3748d2c0aeb740f98f8c2caabad21474, State: Running, Role: LEADER
I20240501 07:59:38.601538 26309 consensus_queue.cc:260] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.21, Last appended by leader: 21, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df5867ab7af4bd0a8229838d80e9d7f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37675 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } }
I20240501 07:59:38.601724 26309 raft_consensus.cc:3037] T ebf9d5a354ac4ef8a8df43802063eb4e P 3748d2c0aeb740f98f8c2caabad21474 [term 6 FOLLOWER]: Advancing to term 7
I20240501 07:59:38.601795 24299 raft_consensus.cc:2396] T 3119928e27b14eb0a68ae0aa0b70059c P 9d371a71c02a4432aca2fed33b2545d5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b8f81da8ae74d3687ef72a2c9ce0888 for term 6 because replica has last-logged OpId of term: 5 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 14.
W20240501 07:59:38.603822 24257 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57338: Illegal state: replica 9d371a71c02a4432aca2fed33b2545d5 is not leader of this config: current role FOLLOWER
I20240501 07:59:38.604957 23893 catalog_manager.cc:5484] T 1056fad606894561bd417d1586cf8032 P 9d371a71c02a4432aca2fed33b2545d5 reported cstate change: term changed from 4 to 6, leader changed from 9d371a71c02a4432aca2fed33b2545d5 (127.0.99.131) to 3b8f81da8ae74d3687ef72a2c9ce0888 (127.0.99.132). New cstate: current_term: 6 leader_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43159 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } }
I20240501 07:59:38.613135 24363 leader_election.cc:302] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b8f81da8ae74d3687ef72a2c9ce0888; no voters: 3748d2c0aeb740f98f8c2caabad21474, 9d371a71c02a4432aca2fed33b2545d5
I20240501 07:59:38.613521 26369 raft_consensus.cc:2726] T 02c7ed9d33204e06a2579bde5ca71960 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 07:59:38.613638 24427 raft_consensus.cc:1226] T ebf9d5a354ac4ef8a8df43802063eb4e P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 7 LEADER]: Rejecting Update request from peer 3748d2c0aeb740f98f8c2caabad21474 for earlier term 6. Current term is 7. Ops: [6.21-6.21]
I20240501 07:59:38.614642 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fb726fc00544504a34a37bf3226f889"
dest_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888"
from {username='slave'} at 127.0.0.1:53256
I20240501 07:59:38.614787 24429 raft_consensus.cc:482] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.614845 24429 raft_consensus.cc:3037] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 5 FOLLOWER]: Advancing to term 6
I20240501 07:59:38.615949 24429 raft_consensus.cc:504] T 8fb726fc00544504a34a37bf3226f889 P 3b8f81da8ae74d3687ef72a2c9ce0888 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3748d2c0aeb740f98f8c2caabad21474" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40707 } } peers { permanent_uuid: "9d371a71c02a4432aca2fed33b2545d5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43575 } } peers { permanent_uuid: "3b8f81da8ae74d3687ef72a2c9ce0888" member_type: VOTER last_known_addr { host: "127.0.99.132"